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PREFACE 


This  is  the  third  volume  of  the  IMIS  Final  Program  Report.  This  volume  documents  the 
results  and  conclusions  of  each  field  test  or  demonstration  conducted  at  Luke  Air  Force  Base. 
Recommendations  and  lessons  learned  collected  during  the  entire  program  (e.g.,  requirements 
analysis,  system  design,  implementation,  integration,  test,  installation,  and  demonstration)  are 
provided.  Finally,  strategies  for  further  implementation  are  included.  The  Debrief  Test  was 
composed  of  real  maintenance  debriefs  and  IMIS  interaction  with  the  production  Core 
Automated  Maintenance  System  (CAMS).  Comparison  of  data  collected  revealed  the  average 
maintenance  debrief  time  using  IMIS  was  less  than  half  the  time  required  when  using  the  current 
method.  Although  debriefs  for  discrepant  aircraft  kept  the  pilots  involved  for  half  a  minute 
longer  than  the  current  method,  total  debrief  time  still  averaged  five  minutes  less  when  using 
IMIS.  The  End-to-End  Demonstration  exercised  the  primary  functions  of  IMIS  using  a  total  of 
32  expediters,  production  superintendents,  maintenance  debriefs,  and  technicians.  Subjective 
data  collected  from  the  participants  identified  IMIS  strengths  as  (1)  ready  availability  of  all 
needed  technical  order  (TO)  data,  (2)  insulation  from  keystroke-by-keystroke  interaction  with 
CAMS,  and  (3)  part-ordering  from  the  job  site.  The  primary  IMIS  weakness  was  insufficient 
speed  during  certain  transactions.  The  Fault  Isolation  Test  successfully  demonstrated  the  IMIS 
concept.  Technicians  were  able  to  complete  fault  isolation  and  repair  problems  with  greater 
accuracy,  in  a  shorter  time,  and  with  fewer  errors  and  fewer  parts  used  in  the  process.  The 
technicians  had  very  little  trouble  using  IMIS  after  a  short  training  session.  Both  the  specialist 
and  airplane  general  (APG)  technicians  were  able  to  perform  the  test  tasks  with  minimal 
difficulty.  In  contrast,  the  APG  technicians  (and  some  specialists)  experienced  significant 
difficulties  in  using  the  paper  TOs.  The  detailed  results  from  the  Fault  Isolation  Test  are 
documented  in  separate  Armstrong  Laboratory  reports  (AL/HR-TP- 1995-0033  and  AL/HR-TP- 
1995-0034). 
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INTEGRATED  MAINTENANCE  INFORMATION  SYSTEM  (IMIS)  RESULTS, 
CONCLUSIONS,  AND  RECOMMENDATIONS 


INTRODUCTION 

This  Final  Program  Report  documents  the  results  of  each  field  test  and  demonstration 
conducted  at  Luke  Air  Force  Base  (AFB)  as  well  as  the  conclusions  that  can  be  drawn  from  these 
results.  In  addition,  recommendations  and  lessons  learned  collected  during  the  entire  program 
(e.g.,  requirements  analysis,  system  design,  implementation,  integration,  test,  installation,  and 
demonstration)  are  provided.  Finally,  recommendations  and  strategies  for  proceeding  with 
further  implementation  are  included. 


RESULTS 

The  results  of  the  Debrief  Test,  End-to-End  Demonstration,  and  Fault  Isolation  Test  are 
summarized  in  the  following  subsections. 


Debrief  Test 

The  Debrief  Test  was  conducted  at  Luke  AFB  from  November  3  through  December  9, 
1993.  Data  was  collected  by  observing  and  timing  actual  debrief  sessions,  some  using  IMIS  and 
the  rest  using  the  current  method.  Four  maintenance  debriefers  participated  to  varying  degrees 
over  the  six- week  period;  only  two  debriefers  were  involved  in  the  majority  of  the  debrief 
sessions.  Information  collected  during  the  sessions  included  start  and  stop  times,  problems 
encountered,  discrepancy  data  (for  use  in  subsequent  tracking  of  work  orders),  and  observations 
or  remarks. 

The  data  for  the  Debrief  Test  was  collected  by  observing  live  debrief  sessions,  therefore, 
the  Debrief  Test  could  not  be  designed  to  provide  a  statistically  valid  test  of  various  hypotheses. 
Instead,  a  quantitative  comparison  of  the  data  was  performed  to  determine  trends  in  the  debrief 
data  collected  using  IMIS  and  the  current  method.  The  data  collected  and  the  results  are 
summarized  in  the  following  subsections. 

IMIS  Debrief  Results 

Forty-five  debrief  sessions  using  IMIS  were  observed.  The  data  collected,  including  the 
amount  of  time  the  pilot  was  present,  the  total  debrief  time,  and  the  number  of  work  orders 
opened,  is  presented  in  Table  1 . 

Aircraft  maintenance  discrepancies  were  reported  as  they  occurred  from  the  scheduled 
aircraft  sorties;  discrepancies  were  reported  in  only  13  of  the  45  IMIS  debrief  sessions.  Such  a 
small  number  of  discrepancies  was  insufficient  to  support  conclusions  regarding  the  quantities  of 
work  orders  opened  and  the  effectiveness  of  the  discrepancy  information  captured. 
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Table  1.  IMIS  Debrief  Results 


MBER 

DATE 

START 

TIME 

PILOT 

DONE 

END 

TIME 

I 

11/9/93 

16:19 

16:33 

16:55 

2 

11/9/93 

16:57 

17:00 

17:00 

3 

11/10/93 

16:18 

16:25 

16:28 

4 

11/10/93 

15:47 

15:50 

15:50 

5 

11/10/93 

15:42 

15:46 

15:46 

6 

11/10/93 

15:31 

15:38 

15:38 

7 

11/10/93 

16:18 

16:20 

16:20 

8 

12/2/93 

14:13 

14:15 

14:15 

9 

12/2/93 

14:28 

14:31 

14:31 

10 

12/2/93 

15:01 

15:03 

15:03 

11 

12/2/93 

15:09 

15:14 

15:16 

12 

12/2/93 

17:32 

17:35 

17:36 

13 

12/2/93 

19:13 

19:18 

19:18 

14 

12/2/93 

19:36 

19:39 

19:39 

15 

12/3/93 

12:32 

12:35 

12:40 

16 

12/3/93 

13:45 

13:48 

13:48 

17 

12/3/93 

13:45 

13:46 

13:47 

18 

12/3/93 

13:30 

13:33 

13:35 

19 

12/6/93 

19:57 

19:59 

19:59 

20 

12/6/93 

20:00 

20:02 

20:02 

21 

12/6/93 

20:12 

20:15 

20:15 

22 

12/7/93 

14:39 

14:41 

14:41 

23 

12/7/93 

15:40 

15:43 

15:46 

24 

12/7/93 

15:03 

15:05 

15:05 

25 

12/7/93 

14:53 

14:56 

14:56 

26 

12/7/93 

19:24 

19:28 

19:28 

27 

12/7/93 

19:47  i 

19:51 

19:59 

28 

12/7/93 

20:13  ’ 

20:15 

20:15 

29 

12/7/93 

15:06 

15:12 

15:18 

30 

12/8/93 

14:50 

14:53 

14:53 

31 

12/8/93 

15:02 

15:12 

15:14 

32 

12/8/93 

17:27 

17:30 

17:30 

33 

12/8/93 

19:28 

19:30 

19:31 

34 

12/8/93 

19:33 

19:36 

19:41 

35 

12/8/93 

19:43 

19:46 

19:46 

36 

12/8/93 

19:48 

19:52 

19:56 

37 

12/8/93 

19:58 

20:04 

20:05 

38 

12/8/93 

20:32 

20:34 

20:34 

39 

12/9/93 

14:11 

14:17 

14:25 

40 

12/9/93 

14:38 

14:41 

14:41 

41 

12/9/93 

14:58 

15:00 

15:00 

42 

12/9/93 

19:19 

19:20 

19:21 

43 

12/9/93 

19:39 

19:52 

20:02 

44 

12/9/93 

20:03 

20:06 

20:06 

45 

12/9/93 

20:11 

20:19 

20:23 

WORK 

ORDERS 


CODE 


PILOT 

TIME 


DEBRIEF 

TIME 


3 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

1 

0 

1 

0 

1 

0 

0 

1 

0 

I 

1 

0 

1 

0 

0 

0 

3 

0 

1 


3 

1 

1 

I 

1 

1 

1 

I 

I 

1 

I 

1 
1 
1 

2 
1 

1 

2 
1 
1 
1 
1 
2 
1 
1 
1 
2 
1 
2 

1 

2 
1 
1 
2 

1 

2 
2 
1 
2 
1 
1 
1 
2 
1 
2 


0:14 

0:36 

0:03 

0:03 

0:07 

0:10 

0:03 

0:03 

0:04 

0:04 

0:07 

0:07 

0:02 

0:02 

0:02 

0:02 

0:03 

0:03 

0:02 

0:02 

0:05 

0:07 

0:03 

0:04 

0:05 

0:05 

0:03 

0:03 

0:03 

0:08 

0:03 

0:03 

0:01 

0:02 

0:03 

0:05 

0:02 

0:02 

0:02 

0:02 

0:03 

0:03 

0:02 

0:02 

0:03 

0:06 

0:02 

0:02 

0:03 

0:03 

0:04 

0:04 

0:04 

0:12 

0:02 

0:02 

0:06 

0:12 

0:03 

0:03 

0:10 

0:12 

0:03 

0:03 

0:02 

0:03 

0:03 

0:08 

0:03 

0:03 

0:04 

0:08 

0:06 

0:07 

0:02 

0:02 

0:06 

0:14 

0:03 

0:03 

0:02 

0:02 

0:01 

0:02 

0:13 

0:23 

0:03 

0:03 

0:08 

0:12 

2 


Minor  problems  were  identified  during  the  IMIS  debrief  sessions.  These  included 
discrepancies  in  the  authored  Fault  Reporting  Manual  (FRM)  data,  recommended  software 
enhancements  (especially  involving  facilitating  error  correction),  and  environmental  difficulties 
resulting  when  inundated  with  pilots  to  be  debriefed.  None  of  these  problems  significantly 
affected  the  outcome  of  the  debrief  sessions. 

Paper  Debrief  Results 

One  hundred  and  eleven  debrief  sessions  using  the  current  paper-based  system  were 
observed.  Data  from  two  of  these  sessions  was  incomplete  or  inconsistent  and  was  discarded, 
leaving  a  total  of  109  data  points.  The  data  collected  can  be  found  in  Table  2. 

Table  2.  Paper  Debrief  Results 


NUMBER 

DATE 

START 

TIME 

PILOT 

DONE 

END 

TIME 

WORK 

ORDERS 

CODE 

PILOT 

TIME 

DEBRIEF 

TIME 

101 

1 1/3/93 

18:57 

19:06 

19:13 

2 

3 

0:09 

MM 

1 1/3/93 

18:59 

19:02 

19:08 

1 

2 

0:03 

1 1/3/93 

19:11 

19:15 

19:18 

0 

1 

0:04 

0:07 

104 

1 1/3/93 

19:11 

19:15 

19:20 

1 

2 

0:04 

0:09 

105 

1 1/3/93 

19:49 

19:51 

20:08 

4 

2 

0:02 

0:19 

106 

1 1/3/93 

19:52 

19:59 

20:11 

0 

1 

0:07 

0:19 

107 

1 1/3/93 

19:57 

20:01 

20:18 

2 

2 

0:04 

0:21 

108 

1 1/3/93 

20:23 

20:28 

20:35 

2 

2 

0:05 

0:12 

109 

11/3/93 

20:14 

20:16 

20:20 

1 

0:02 

0:06 

no 

11/4/93 

14:05 

14:11 

14:17 

2 

0:06 

0:12 

111 

11/4/93 

14:33 

14:35 

14:36 

1 

0:02 

0:03 

112 

1 1/4/93 

14:33 

14:37 

14:40 

1 

0:04 

0:07 

113 

11/4/93 

15:04 

15:09 

15:09 

1 

0:05 

0:05 

114 

11/4/93 

15:09 

15:13 

15:33 

2 

0:04 

0:24 

115 

11/4/93 

15:09 

15:15 

15:20 

1 

0:06 

0:11 

116 

11/4/93 

15:09 

15:16 

15:25 

3 

3 

0:07 

0:16 

117 

11/4/93 

15:14 

15:17 

15:23 

1 

0:03 

0:09 

118 

11/4/93 

15:14 

15:18 

15:28 

1 

0:04 

0:14 

119 

11/4/93 

15:24 

15:27 

15:33 

1 

0:03 

0:09 

120 

11/4/93 

15:25 

15:26 

15:33 

1 

0:01 

0:08 

121 

1 1/4/93 

19:02 

19:05 

19:06 

1 

0:03 

0:04 

122 

1 1/4/93 

19:02 

19:07 

19:13 

1 

0:05 

0:11 

123 

11/4/93 

19:14 

19:16 

19:19 

1 

0:02 

0:05 

124 

11/4/93 

19:16 

19:20 

19:21 

1 

0:04 

0:05 

125 

11/4/93 

19:35 

19:38 

19:42 

1 

2 

0:03 

0:07 

126 

11/4/93 

19:35 

19:40 

19:43 

1 

3 

0:05 

0:08 

3 


Table  2.  Continued 


NUMBER 

DATE 

START 

TIME 

PILOT 

DONE 

END 

TIME 

WORK 

ORDERS 

CODE 

PILOT 

TIME 

DEBRIEF 

TIME 

127 

1 1/4/93 

19:41 

19:45 

20:04 

1 

2 

0:04 

0:23 

128 

1 1/4/93 

19:42 

19:44 

19:47 

0 

1 

0:02 

0:05 

129 

1 1/4/93 

20:23 

20:26 

20:26 

0 

1 

0:03 

0:03 

130 

1 1/5/93 

9:21 

9:26 

9:26 

0 

1 

0:05 

0:05 

131 

11/5/93 

9:21 

9:26 

9:32 

1 

2 

0:05 

0:11 

132 

1 1/5/93 

14:31 

14:33 

14:35 

1 

2 

0:02 

0:04 

133 

1 1/5/93 

14:33 

14:39 

14:45 

1 

2 

0:06 

0:12 

134 

11/5/93 

14:59 

15:02 

15:04 

0 

1 

0:03 

0:05 

135 

11/5/93 

14:34 

14:38 

14:56 

2 

2 

0:04 

0:22 

136 

11/5/93 

15:27 

15:31 

15:37 

1 

2 

0:04 

0:10 

137 

11/5/93 

14:37 

14:40 

15:00 

2 

2 

0:03 

0:23 

138 

1 1/8/93 

15:37 

15:44 

16:00 

1 

2 

0:07 

0:23 

139 

11/8/93 

16:58 

17:00 

17:16 

0 

1 

0:02 

0:18 

140 

11/8/93 

17:16 

17:18 

17:19 

0 

1 

0:02 

0:03 

141 

11/8/93 

17:00 

17:02 

17:23 

0 

1 

0:02 

0:23 

142 

11/8/93 

15:30 

15:36 

15:45 

1 

2 

0:06 

0:15 

143 

11/8/93 

16:58 

17:00 

17:01 

0 

1 

0:02 

0:03 

144 

1 1/8/93 

19:59 

20:03 

20:08 

2 

3 

0:04 

0:09 

145 

.  11/8/93 

20:00 

20:05 

20:12 

1 

2 

0:05 

0:12 

146 

1 1/8/93 

20:17 

20:19 

20:21 

0 

1 

0:02 

0:04 

147 

1 1/8/93 

20:44 

20:45 

20:47 

0 

1 

0:01 

0:03 

148 

11/8/93 

20:44 

20:50 

20:58 

2 

3 

0:06 

0:14 

149 

1 1/8/93 

21:09 

21:12 

21:13 

0 

1 

0:03 

0:04 

150 

11/9/93 

16:57 

17:00 

17:14 

1 

3 

0:03 

0:17 

151 

11/9/93 

16:09 

16:17 

16:20 

1 

2 

0:08 

0:11 

152 

1 1/9/93 

16:06 

16:09 

16:14 

0 

1 

0:03 

0:08 

153 

11/9/93 

15:57 

16:00 

16:01 

0 

1 

0:03 

0:04 

154 

11/9/93 

15:37 

15:40 

15:42 

0 

1 

0:03 

0:05 

155 

11/9/93 

15:34 

15:38 

15:40 

0 

1 

0:04 

0:06 

156 

11/9/93 

15:35 

15:37 

15:46 

0 

1 

0:02 

0:11 

157 

11/9/93 

15:34 

15:36 

15:46 

2 

3 

0:02 

0:12 

158 

11/9/93 

19:02 

19:05 

19:07 

1 

3 

0:03 

0:05 

159 

1 1/9/93 

19:07 

19:11 

19:17 

1 

3 

0:04 

0:10 

160 

11/9/93 

19:21 

19:31 

19:32 

0 

1 

0:10 

0:11 

161 

11/9/93 

20:00 

20:03 

20:03 

0 

1 

0:03 

0:03 

162 

1 1/9/93 

20:12 

20:23 

20:42 

3 

2 

0:11 

0:30 

163 

11/9/93 

20:20 

20:23 

20:37 

0 

1 

0:03 

0:17 

166 

11/10/93 

15:32 

15:37 

15:42 

1 

2 

0:05 

0:10 

167 

11/10/93 

15:34 

15:37 

16:02 

0 

1 

0:03 

0:28 

168 

11/10/93 

15:36 

15:38 

16:04 

0 

1 

0:02 

0:28 

169 

11/10/93 

16:47 

16:53 

16:56 

1 

3 

0:06 

0:09 

170 

11/10/93 

16:56 

16:58 

17:10 

0 

1 

0:02 

0:14 

171 

11/30/93 

13:41 

^  13:45 

13:48 

0 

1 

0:04 

0:07 

172 

1 1/30/93 

13:41 

13:46 

13:49 

1  1 

2 

0:05 

0:08 

173 

1 1/30/93 

13:53 

13:56 

13:56 

0 

1 

0:03 

0:03 

174 

1 1/30/93 

14:16 

14:18 

14:20 

0 

1 

0:02 

0:04 
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Table  2.  Continued 


NUMBER 

DATE 

START 

TIME 

PILOT 

DONE 

END 

TIME 

WORK 

ORDERS 

CODE 

PILOT 

TIME 

DEBRIEF 

TIME 

175 

1 1/30/93 

13:53 

14:22 

1 

2 

0:10 

176 

1 1/30/93 

14:15 

14:22 

14:36 

1 

2 

0:07 

177 

1 1/30/93 

14:22 

14:27 

14:37 

1 

2 

0:05 

178 

1 1/30/93 

14:34 

14:53 

2 

0:04 

179 

1 1/30/93 

14:17 

14:37 

15:21 

3 

0:20 

180 

11/30/93 

14:59 

15:27 

3 

0:05 

0:28 

181 

1 1/30/93 

17:13 

0 

1 

0:01 

0:05 

182 

1 1/30/93 

19:12 

1 

2 

0:10 

0:38 

183 

1 1/30/93 

19:23 

19:27 

19:44 

1 

2 

0:04 

0:21 

184 

1 1/30/93 

19:46 

19:52 

mSm 

1 

0:06 

0:16 

185 

1 1/30/93 

19:43 

19:55 

919 

1 

0:12 

0:28 

186 

11/30/93 

20:17 

1 

0:04 

0:17 

187 

11/30/93 

19:19 

19:39 

3 

2 

0:20 

1:23 

188 

12/1/93 

13:49 

13:49 

1 

2 

0:09 

0:09 

189 

12/1/93 

■39 

13:47 

13:56 

1 

2 

0:04 

0:13 

190 

12/1/93 

13:42 

13:47 

13:57 

1 

2 

0:05 

0:15 

191 

12/1/93 

13:44 

13:47 

1 

0:03 

0:24 

192 

12/1/93 

13:46 

13:47 

0 

1 

0:01 

0:24 

193 

12/1/93 

13:52 

13:58 

14:12 

1 

2 

0:06 

0:20 

194 

12/1/93 

14:22 

14:26 

14:31 

1 

0:04 

0:09 

195 

12/1/93 

14:32 

14:34 

14:36 

1 

0:02 

0:04 

196 

12/1/93 

14:33 

14:38 

14:41 

1 

2 

0:05 

0:08 

197 

12/1/93 

14:36 

mSm 

14:43 

0 

1 

0:04 

0:07 

198 

12/1/93 

14:55 

■39 

■B9 

1 

2 

0:05 

0:09 

199 

12/1/93 

14:56 

14:59 

■39 

0 

1 

0:03 

0:13 

200 

12/1/93 

14:56 

15:11 

2 

2 

0:04 

0:15 

201 

12/1/93 

15:44 

■93 

mmm 

1 

3 

0:06 

0:16 

202 

12/1/93 

18:56 

18:59 

■91 

0 

1 

0:03 

0:04 

203 

12/1/93 

18:56 

■39 

3 

0:04 

0:09 

204 

12/1/93 

19:12 

19:15 

1 

0:06 

0:09 

205 

12/1/93 

19:10 

19:16 

3 

0:06 

0:12 

206 

12/1/93 

19:18 

19:27 

19:32 

3 

0:09 

0:14 

207 

12/1/93 

19:30 

19:34 

19:36 

0 

1 

0:04 

0:06 

208 

12/1/93 

19:38 

19:42 

19:45 

0 

1 

0:04 

0:07 

209 

12/1/93 

19:39 

19:42 

19:50 

0 

1 

0:03 

0:11 

210 

12/1/93 

19:36 

19:42 

19:58 

2 

2 

0:06 

0:22 

211 

12/2/93 

19:44 

19:47 

0 

1 

0:03 

0:06 

The  problems  encountered  during  some  of  these  debriefs  centered  primarily  on  the  Core 
Automated  Maintenance  System  (CAMS).  In  one  case,  CAMS  rejected  a  debrief  because  the 
maintenance  debriefer  had  not  been  notified  of  a  tail  number  swap.  Several  other  sessions 
reported  that  CAMS  is  slow,  takes  an  extremely  long  time  between  screens,  and  does  not  accept 
input.  These  problems  added  to  the  overall  debrief  times  because  debriefer  interaction  was 
required. 
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Comparison  of  Debrief  Results 


A  comparison  of  the  data  collected  during  the  Debrief  Test  revealed  some  clear  trends. 
Again,  it  is  important  to  note  that  the  data  was  not  collected  in  accordance  with  a  statistically 
valid  experimental  design;  therefore,  a  comparison  with  any  level  of  statistical  significance  is  not 
possible.  A  summary  of  the  comparison  data  can  be  found  in  Table  3,  which  lists  average  times 
and  sample  sizes  for  each  categorization  of  the  data. 


Table  3.  Comparison  of  Results 


IMIS 

Current  Method 

Debrief  Time  (All) 

5:56 

(45  debriefs) 

13:26 

(109) 

Pilot  Time  (All) 

3:57 

(45) 

4:36 

(109) 

Debrief  Time  (Code  1  Only) 

3:15 

(32) 

9:36 

(56) 

Pilot  Time  (Code  1  Only) 

2:58 

(32) 

3:26 

(56) 

Debrief  Time  (Code  2,3) 

12:32 

(13) 

17:29 

(53) 

Pilot  Time  (Code  2,3) 

6:23 

(13) 

5:50 

(53) 

Debrief  Time  (1  Work  Order) 

9:27 

(11) 

14:05 

(37) 

Pilot  Time  (1  Work  Order) 

5:05 

(11) 

5:19 

(37) 

Debrief  Time  (>1  Work 
Order) 

29:30 

(2) 

25:23 

(16) 

- — - - - i 

Pilot  Time  (>1  Work  Order) 

13:30 

(2) 

7:00 

(16) 

Overall,  the  average  dehrief  time  using  IMIS  was  less  than  half  that  using  the  current 
method  (5:56  vs.  13:26).  The  average  pilot  time,  considering  all  data,  was  also  less,  although  the 
difference  was  not  nearly  as  dramatic  (3:57  vs.  4:36). 


When  considering  only  Code  1  debriefs  (i.e.,  no  discrepancies  reported),  the  average 
debrief  time  is  reduced  by  a  factor  of  two-thirds  when  using  IMIS  (3:15  vs.  9:36).  In  these  cases, 
the  completion  of  the  IMIS  debrief  requires  only  an  additional  17  seconds  after  the  pilot  is  done, 
as  compared  to  the  current  method,  for  which  an  additional  six  minutes  is  required. 

In  debriefs  where  discrepancies  were  reported  and  work  orders  were  opened,  the  average 
pilot  time  was  not  less  when  using  IMIS  compared  to  using  the  current  method  (6:23  vs.  5:50) 
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because  the  pilot  was  present  as  each  work  order  was  opened  and  as  the  appropriate  entry  from 
the  FRM  data  was  selected.  The  average  overall  debrief  time  for  these  debriefs  was  still  less 
(12:32  vs.  17:29). 

The  times  were  similar  when  only  a  single  work  order  was  opened.  However,  due  to  the 
small  number  of  debriefs  where  multiple  work  orders  were  opened  using  IMIS  (only  2),  no 
comparisons  can  be  made. 

User  Feedback 

The  user  feedback  collected  by  the  maintenance  debriefs  after  the  completion  of  the 
Debrief  Test  was  valuable.  The  feedback  indicated  that  the  most  liked  aspect  of  the  IMIS 
debriefing  function  is  that  conducting  a  debrief  session  for  a  Code  1  (no  discrepancies)  aircraft 
(A/C)  is  extremely  fast,  aided  especially  by  the  pre-filling  of  data  elements  from  the  flying 
schedule  and  by  the  availability  of  listers  to  select  data  and  minimize  data  entry  errors. 
Additionally,  the  enhanced  question  sets  used  when  opening  a  work  order  were  helpful  to  the 
debriefers  when  they  were  not  qualified  for  or  familiar  with  a  particular  discrepant  A/C  system. 
Thus,  users  believed  that,  with  the  enhanced  question  sets  to  assist  in  the  debrief  process,  anyone 
can  debrief  a  system.  However,  the  enhanced  question  sets  were  also  viewed  as  a  somewhat 
negative  factor  because  they  require  additional  time  in  the  debrief  of  the  pilot  for  a  Code  2  or 
Code  3  A/C. 

The  maintenance  debriefers  made  several  recommendations  regarding  functionality 
enhancements  which  would  make  the  system  even  better,  such  as  the  ability  to  interactively 
update  the  enhanced  question  sets.  The  debriefers  also  noted  that  they  must  frequently  update 
the  debrief  results  in  CAMS  when  new  or  revised  information  becomes  available;  by  providing 
the  capability  to  edit  debrief  data  in  IMIS  after  it  has  been  accepted,  the  debrief  process  would 
become  more  efficient.  Flying  schedule  and  tail  number  swaps  occurred  frequently  during  the 
Debrief  Test,  and  IMIS  demonstrated  a  limited  ability  to  respond  to  these  last-minute  changes. 
The  debriefers  believe  it  would  be  desirable  to  provide  the  capability  to  debrief  a  tail  number 
which  is  not  on  the  list  of  undebriefed  sorties  rather  than  wait  for  the  change  to  be  entered  into 
CAMS  and  propagated  to  IMIS  manually. 

End-to-End  Demonstration 

The  IMIS  End-to-End  Demonstration  was  a  field  demonstration  of  the  primary  functions 
of  IMIS.  This  demonstration  was  conducted  on  F-16  Block  40/42  A/C  assigned  to  the  310th 
Fighter  Squadron  (FS)  at  Luke  AFB,  AZ,  from  1  June  through  30  June  1994.  It  was  intended  to 
illustrate  to  users  the  overall  IMIS  concept  by  using  the  system  to  support  a  series  of  typical 
maintenance  scenarios  in  an  operational  environment,  under  structured  conditions.  The 
demonstration  showed  system  functional  capabilities  in  all  primary  IMIS  functional  areas: 
debrief,  diagnostics,  electronic  TOs,  work  order  generation/close-out,  and  flightline  management 
support. 
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The  basic  objectives  of  the  End-to-End  Demonstration  were  achieved  with  the  validation 
of  the  IMIS  concept.  The  demonstration  showed  that  IMIS  effectively  provides  information  the 
managers  and  technicians  require,  provides  information  in  a  way  that  is  acceptable  to  users,  and 
is  something  users  want.  The  participants  also  provided  valuable  feedback,  including 
suggestions  for  improving  the  system  and  for  improving  the  content  and  organization  of  data 
presented  by  the  system. 

A  total  of  32  participants,  including  expediters,  production  superintendents,  maintenance 
debriefers,  and  technicians,  completed  the  exercise.  All  the  participants  either  were  currently 
performing  in  the  job  they  represented  or  had  recent  experience  in  that  position.  Data  was 
collected  from  the  participants  using  three  different  tools;  an  exit  questionnaire,  an  automated 
questionnaire  that  collected  opinions  on  IMIS  functional  characteristics,  and  the  National 
Aeronautics  and  Space  Administration  (NASA)  Task  Load  Index  (TLX). 


Exit  Questionnaire  Results 

The  exit  questionnaire  contains  three  direct  questions  and  a  place  for  the  respondent  to 
add  written  comments.  The  three  questions  are: 

1 .  What  did  you  like  about  IMIS  as  an  aid  to  help  you  do  your  job? 

2.  What  did  you  dislike  about  IMIS? 

3.  What  changes  would  you  make  to  improve  the  system? 

The  participants  recognized  that  the  current  demonstration  system  is  intended  only  for  use 
in  evaluating  the  concept  of  IMIS  as  a  tool  to  establish  requirements  for  a  system  to  be  developed 
for  operational  use.  Consequently,  they  were  able  to  evaluate  the  system  on  its  potential. 
Overall  responses  to  the  questionnaire  indicate  a  positive  reaction  to  the  IMIS  concept.  Many 
responses  were  very  laudatory.  The  only  exceptions  were  comments  directed  toward  weaknesses 
in  the  demonstration  system. 

The  following  is  a  summary  of  the  participants'  responses  to  the  question,  "What  did  you 
like  about  IMIS  as  an  aid  to  help  you  do  your  job?" 

a.  The  large  amount  of  information  available  through  IMIS.  Participants  were 
impressed  by  the  fact  that,  with  a  few  key  strokes,  they  could  quickly  access 
information  which  they  would  normally  have  to  track  down  manually. 

b.  The  currency  of  the  available  information.  Participants  liked  the  fact  that  not  only  are 
schedules  and  similar  information  kept  current  but  that  key  people  are  notified  of 
changes. 

c.  Not  having  to  v/ork  with  CAMS  to  extract  or  input  data.  IMIS  automatically 
interfaces  with  CAMS  and  does  not  require  the  user  to  manually  enter  data  into 
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CAMS.  This  was  seen  as  a  great  advantage,  in  that  this  automatic  interface  could 
save  time  and  provide  more  accurate  information,  for  all  maintenance  personnel  and 
the  maintenance  information  management  system. 

d.  Ready  access  to  TOs.  The  availability  of  all  required  TOs  on  the  Portable 
Maintenance  Aid  (PMA)  was  seen  as  a  benefit  because  the  technician  will  not  have  to 
carry  a  large  number  of  TOs  to  the  flightline  or  return  to  the  shop  for  other  TOs  that 
may  be  needed. 

e.  Parts  ordering.  Both  technicians  and  supervisors  liked  being  able  to  access  parts 
availability  information  rapidly  through  the  PMA.  They  also  liked  the  provision  of 
notifying  the  production  superintendent  of  a  part  order  request  and  for  getting  his  or 
her  authorization  for  ordering  the  part. 

The  following  is  a  summary  of  the  participants'  responses  to  the  question,  "What  did  you 
dislike  about  IMIS?" 

a.  Speed.  Almost  all  participants  mentioned  speed  as  a  problem  for  most  maintenance 
management  information  support  functions  (times  ranged  from  35  to  90  seconds  for 
data  input  and  updating).  However,  respondents  differed  in  their  perception  of  the 
impact  of  speed.  Some  believed  it  could  be  quite  important  (". . .  30  seconds  is  a  long 
time  when  'the  man'  is  looking  over  your  shoulder").  Others  saw  the  issue  from  a 
different  perspective  (".  .  .  the  time  doesn't  seem  so  bad  when  you  consider  that  it 
would  take  me  15  or  20  minutes  to  track  the  same  information  today").  Another 
noted  that  "the  availability  of  the  information  makes  up  for  the  slowness  of  the 
system." 

b.  Usability  issues.  There  were  several  human/computer  interaction  procedures  which 
some  technicians  found  inconsistent.  The  primary  problem  was  the  use  of  the  FI  and 
SELECT  keys.  In  some  cases,  a  selection  was  made  with  the  FI  key;  in  other  cases 
the  SELECT  key  was  used.  The  problem:  it  was  not  always  obvious  which  key  to 
use.  The  second  most  frequently  mentioned  usability  problem  was  the  confusion 
between  the  FI  and  F8  keys  when  accessing  TOs.  IMIS  uses  FI  as  the  convention  to 
approve  data  and  continue  with  data  processing.  The  TO  presentation  software, 
however,  uses  F8  to  go  to  the  next  presentation.  After  using  IMIS  for  a  while,  the 
user  would  become  accustomed  to  pressing  FI  to  proceed.  When  using  the  TO 
presentation,  the  user  then  had  to  press  F8  to  proceed.  It  took  time  for  the  user  to 
become  accustomed  to  this  change  and  the  corresponding  adjustment  when  returning 
to  IMIS. 

c.  PMA  keypad.  Many  participants  encountered  difficulty  using  the  PMA  keypad.  The 
keypad  requires  a  firm  press;  consequently,  it  was  easy  to  think  that  it  had  been 
activated  when  it  had  not.  Also,  the  PMA  did  not  always  respond  immediately  when 
the  key  was  pressed,  depending  upon  the  amount  of  background  processing  occurring. 
When  this  happened,  there  was  a  natural  tendency  to  immediately  press  the  key  again, 
which  occasionally  caused  the  user  to  miss  a  step  or  make  a  wrong  selection. 
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d.  Reliability.  The  PMA  was  prone  to  software  failures  during  the  End-to-End 
Demonstration.  Most  participants  experienced  at  least  one  PMA  crash  during  the 
session.  Some  of  these  failures  were  caused  by  known  problems,  while  others  were 
more  difficult  to  identify.  These  problems  were  documented  and  most  were  corrected 
before  the  Fault  Isolation  Test  was  started. 

Characteristics  Questionnaire  Results 

The  characteristics  questionnaire  consisted  of  97  questions  designed  to  measure 
participants'  evaluations  of  various  characteristics  of  the  IMIS  demonstration  system.  The 
questionnaire  required  participants  to  indicate  the  degree  of  agreement  with  a  statement  about  an 
IMIS  characteristic.  The  response  was  made  on  a  seven-point  scale.  The  complete  questionnaire 
covered  all  key  features  and  characteristics  of  IMIS.  The  rating  scale  used  was  from  1  (very 
positive)  to  7  (very  negative).  Participants  only  responded  to  questions  on  features  which  they 
had  experienced  during  the  demonstration.  Consequently,  from  the  total  of  97  questions,  41 
were  relevant  during  the  End-to-End  Demonstration.  Some  of  these  41  questions  were 
determined  to  have  been  incorrectly  structured  and  were  subsequently  eliminated,  reducing  the 
total  number  of  questions  to  35. 

Questionnaire  results  indicate  that  responses  are  generally  consistent  with  respondents' 
answers  to  the  exit  questionnaire.  Overall,  responses  were  positive.  The  total  overall  rating  for 
all  responses  was  3.01.  Nineteen  of  the  questions  were  classified  as  either  very  positive  or 
positive.  Three  items  were  classified  as  negative  (none  were  classified  as  very  negative). 
Ratings  for  the  remaining  13  questions  were  classified  as  neutral.  A  review  of  all  the  responses 
showed  that  the  items  receiving  positive  ratings  include  those  concerning  the  manner  in  which 
information  was  presented  on  the  PMA,  procedures  for  accessing  information,  and  techniques  for 
interacting  with  the  system.  Most  of  the  negative  ratings  related  to  the  specific  characteristics  of 
the  demonstration  PMA,  specifically  the  responsiveness  of  the  keypad  and  the  reliability  of  the 
Radio  Frequency  (RF)  link.  Specific  features  that  were  most  favorably  rated  and  areas  identified 
as  needing  improvement  are  listed  below. 

The  following  features  received  the  highest  positive  ratings  (between  1.0  and  2.0). 

a.  Function  keys  were  useful. 

b.  Training  received  was  thorough. 

c.  Time  permitted  to  become  familiar  with  the  PMA  and  its  functions  was  adequate. 

d.  Spacing  between  the  keys  on  the  keypad  was  acceptable. 

e.  Turning  on/off  the  RF  devices  from  the  menu  was  easy. 

f  The  size  of  the  keys  on  the  expediter's  detachable  PMA  keyboard  was  adequate. 
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The  following  features  received  the  highest  negative  ratings  (between  5.0  and  6.0).  It 
should  be  noted  that  no  characteristic  was  rated  between  6.0  and  7.0. 

a.  Determining  how  to  proceed  through  the  PMA  screens  was  somewhat  difficult. 

b.  Response  time  after  pressing  keys  was  somewhat  slow. 

c.  Sending  messages  on  the  RF  modem  was  somewhat  unreliable. 

The  list  of  all  35  questions  and  their  average  scores  is  provided  in  Table  4. 

NASA  TLX  Results 

The  NASA  TLX  is  a  multidimensional  workload  assessment  tool  designed  to  provide  a 
measure  of  the  workload  imposed  by  a  given  job/work  situation.  The  index  is  composed  of 
weighted  subscores  of  ratings  on  six  factors  which  are  believed  to  contribute  to  workload.  The 
factors  are  mental  demands,  physical  demands,  temporal  demands,  own  performance,  effort,  and 
frustration.  The  NASA  TLX  produces  a  workload  measure  for  each  factor  plus  a  cumulative 
index  of  workload.  Each  index  can  range  from  0  to  100  and  represents  a  measure  of  the 
workload  imposed  by  the  job/work  situation. 

It  is  possible  to  measure  the  relative  workload  imposed  by  two  job/work  situations  by 
having  participants  complete  the  NASA  TLX  for  the  same  job  performed  under  different  work 
situations.  This  approach  was  followed  for  the  End-to-End  Demonstration  to  provide  an  initial 
indication  on  the  impact  of  the  IMIS  on  the  workload  of  maintenance  personnel  versus  the 
current  paper-based  system.  Each  participant  in  the  End-to-End  Demonstration  completed  the 
NASA  TLX  twice:  once  with  the  current  paper-based  methods  of  doing  the  job  as  the  frame  of 
reference  and  once  with  IMIS  as  the  frame  of  reference.  The  paper-based  ratings  were  based 
upon  how  technicians  do  the  job  now  (based  upon  retrospection).  The  IMIS-based  ratings  were 
made  immediately  following  the  use  of  IMIS  (based  on  current  experience). 

Complete  NASA  TLX  ratings  were  made  by  16  of  the  32  subjects  (data  on  the  remaining 
16  subjects  was  lost  due  to  computer  malfunctions  and  administration  irregularities).  Analysis  of 
the  ratings  yielded  a  mean  TLX  workload  index  of  62.75  for  the  paper-based  condition  and  44.44 
for  the  IMIS  condition.  The  difference  between  means  is  statistically  significant  at  the  0.9999 
confidence  level. 

The  results  suggest  that  for  this  End-to-End  Demonstration  IMIS  significantly  reduced 
the  workload  experienced  by  expediters  and  production  superintendents.  However,  caution 
should  be  used  in  interpreting  the  NASA  TLX  results  because  they  are  based  upon  a  relatively 
small  sample,  and  the  paper-based  ratings  are  based  on  retrospection  rather  than  immediate 
experience.  It  should  be  noted  that  these  findings  are  in  the  same  direction  as  and  reinforce  early 
results  from  previous  field  tests  showing  more  rapid  job  performance  when  electronic  technical 
data  is  used  in  place  of  paper-based  technical  data.  However,  these  conclusions  remain  to  be 
fully  validated  when  tested  in  an  unconstrained  operational  environment. 
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Table  4.  Characteristics  Questionnaire  Results 


Characteristic  (Response) _ Average  Score 


When  performing  maintenance,  the  size  of  the  text  was  2.04 

(Easy  to  read  -  Difficult  to  read) _ 


Highlighting  information  on  the  screen  made  tasks  2.00 

(Easier  -  Harder) _ 

Accessing  screens  required  to  perform  the  task  was  (Easy  2.64 

-  Difficult)  _ _ 


Determining  how  to  proceed  through  the  PMA  screens  in  5.90 

order  to  support  your  task  was  (Easy  -  Difficult) _ 


Message  lines  (informing  you  how  to  proceed  on  to  the  2.29 

next  piece  of  information)  were  (Effective  -  Ineffective)  _ 


Function  keys  (FI  through  F8)  were  (Useful  -  Not  1 .48 

Useful) _ _ _ 

Use  of  icons  was  (Easy  -  Hard) _ 2.47 


Symbols  chosen  for  icon  pictures  were  (Effective  -  Not  2.05 

Effective) _ 


Reliability  of  the  IMIS  machine  that  you  worked  with  3 .40 

was  (Very  Reliable  -  Unreliable)  _ 

The  training  you  received  was  (Thorough  -  Not  1 .50 

Thorough)  _ _ 


Time  permitted  to  become  familiar  with  the  PMA  and  its  1 .86 

functions  was?  (Adequate  -  Inadequate) _ 


The  weight  of  the  PMA  was  (Heavy  —  Light) _ 4.63* 


Inverse  value  of  3.37  used  to  allow  lower  numbers  to  reflect  positive  characteristic  (lighter  weight  rather  than 
heavier  weight). 


Table  4.  Continued 


The  width  and  length  (general  shape)  of  the  PMA  were  2.10 

(Acceptable  -  Not  Acceptable) 

The  prop  or  handle  was  (Effective  -  Not  Effective)  2.20 

The  ruggedness  of  the  keys  on  the  keypad  appeared  to  be  2.90 

(Acceptable  -  Not  Acceptable) 

Spacing  between  keys  on  the  keypad  was  (Acceptable  -  1 .67 

Not  Acceptable) 

Pressing  and  activating  keys  on  the  keypad  was  (Easy-  3.80 

Hard) _ 

Moving  the  cursor  around  the  screen  using  the  PMA  3.35 

arrow  keys  was  (Easy -Hard) _ 

The  cursor  moved  where  you  thought  it  should  (Always  3.13 

-  Never) 

Moving  the  pointer  around  the  screen  using  the  PMA  3 .05 

thumb-knob  was  (Easy  -  Hard) 

Response  time  after  pressing  keys  was  (Fast  Enough  -  5.22 

Too  Slow) 

The  size  of  the  PMA  screen  for  displaying  text  was  2.05 

(Adequate  -  Not  Adequate) 

The  size  of  the  PMA  screen  for  displaying  graphical  2.24 

information  was  (Adequate  -  Not  Adequate) _ 

The  PMA  screen  made  information  (Easy  to  See  -  Hard  2.81 

to  See) 

Ability  to  see  and  read  the  screen  contents  from  various  4.64 

angles  was  (Easy -Hard) _ 

The  screen  was  readable  at  (Many  Angles  -  Limited  3.70 

Angles) 
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Table  4.  Concluded 


Glare  on  the  screen  affected  performance  on  the  task 
(Not  At  All  -  A  Great  Deal) 

4.09 

The  back  light  was  (Helpful  -  Not  Helpful) 

4.93 

The  brightness  of  the  back  light  was  (Good  -  Bad) 

3.29 

Sending  messages  on  the  RF  modem  was  (Reliable  - 
Unreliable) 

5.94 

Turning  off  the  RF  device  from  the  menu  was  (Easy  - 
Difficult) 

1.44 

Messages  sent  by  RF  link  were  responded  to  (Quickly  - 
Slowly) 

3.88 

The  IMIS  automatic  status  updating  would  reduce  the 
chatter  on  "bricks"  (A  Great  Deal  -  Not  At  All) 

2.00 

The  size  of  the  keys  on  the  expediter's  detachable 
keyboard  was  (Adequate  -  Not  Adequate) 

1.83 

Limited  descriptions  in  the  discrepancy  field  of  the  work 
order  form  had  (No  Impact  -  Negative  Impact) 

4.11 

Other  Activities/Results 

The  PMA  RF  capability  was  tested  by  transmitting  and  receiving  messages  from  the 
PMA  to  the  IMIS  base  antenna  located  on  the  310th  FS  hangar  (Building  913).  Transmissions 
were  made  at  distances  of  up  to  2500  feet  (762  meters),  sufficient  to  cover  the  310th  FS  A/C 
parking  area.  No  problems  were  encountered  in  transmitting  messages  at  these  distances.  In 
addition,  the  RF  capability  was  tested  with  up  to  four  PMAs  sending  and  receiving.  No 
problems  were  encountered.  However,  it  became  apparent  that  the  more  PMAs  in  operation,  the 
slower  the  transfer  of  messages  between  the  PMAs  and  the  base  station.  Also,  using  the  RF 
slows  down  non-RF-related  processes  on  the  PMAs  because  the  PMA  has  to  interrupt  ongoing 
processes  to  receive  messages  and  updates. 

The  PMAs  were  installed  in  the  expediter  vehicle  and  tested  without  problems.  A  PMA 
was  also  installed  in  the  production  superintendent's  golf  cart  but  was  not  used  for  transmissions. 
Some  minor  modifications  for  the  mounting  racks  were  identified. 
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The  PMA  was  not  formally  tested  for  heat  tolerance  during  the  End-to-End 
Demonstration.  However,  it  was  used  under  high-temperature  conditions.  It  was  used  in  an 
expediter  truck  without  air  conditioning  in  ambient  temperatures  up  to  117°F  (47°C).  In 
addition,  it  was  used  in  a  hangar  environment  for  several  hours  per  session  in  temperatures  up  to 
1 10°F  (43°C).  No  problems  were  encountered  with  the  PMAs  due  to  heat.  This  is  a  significant 
finding  because  heat  tolerance  had  previously  been  a  concern. 

Fault  Isolation  Test 

The  detailed  results  from  the  Fault  Isolation  Test  are  documented  in  separate  Armstrong 
Laboratory  reports  (AL/HR-TP- 1995-0033  and  AL/HR-TP-1 995-0034). 


CONCLUSIONS 

The  conclusions  reached  as  part  of  the  IMIS  field  tests  and  demonstrations  are 
documented  in  the  following  subsections. 


Debrief  Test 

The  debrief  process  was  much  more  efficient  using  IMIS  as  compared  with  the  current 
method  because  of  the  smaller  number  of  screens  accessed  and  because  the  user's  direct  interface 
with  CAMS  had  been  eliminated.  This  was  especially  true  for  the  Code  1  debrief  sessions  during 
which  no  work  orders  were  opened.  In  addition,  the  amount  of  time  the  pilot  was  involved  did 
not  increase,  despite  the  fact  that  the  maintenance  debriefer  entered  the  information  into  IMIS 
with  the  pilot  present  (in  contrast  to  the  current  system,  where  the  debriefer  often  waits  until  later 
to  enter  the  information  into  CAMS). 

The  users  also  preferred  the  IMIS  user  interface  features  like  pre-filled  data  and  listers. 
The  enhanced  question  sets  also  provided  the  capability  for  any  maintenance  debriefer, 
regardless  of  experience  level  or  Air  Force  Specialty  Code  (AFSC),  to  ask  technical  questions 
about  the  discrepant  system  and  enter  the  descriptive  discrepancy  information  based  on  the  pilot's 
observations. 


End-to-End  Demonstration 

The  End-to-End  Demonstration  showed  that  IMIS  is  capable  of  providing  maintenance 
persormel  with  both  the  management  and  technical  information  they  require.  Both  the  quantity 
and  the  currency  of  information  exceed  the  data  currently  available  to  them.  However,  the 
system  response  times  and  reliability  need  to  be  improved  in  order  to  enhance  user  efficiency. 

The  ability  to  enter  data  into  a  single  system  and  have  that  system  send  the  data  to  the 
necessary  legacy  databases  is  a  very  desirable  feature.  Eliminating  the  unique  database  interfaces 
will  reduce  the  time  the  maintenance  personnel  spend  interacting  with  the  legacy  databases. 
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A  seamless  interface  between  the  various  software  processes  must  be  developed.  The 
transition  from  the  IMIS  to  the  TO  Presentation  (TO  Present)  software  was  apparent  to  the  user 
and  was  magnified  by  the  differences  in  the  user  interface.  The  user  interface  must  be  made 
consistent  throughout  the  system. 


Fault  Isolation  Test 

The  field  test  successfully  demonstrated  the  feasibility  of  the  IMIS  concept  and  the  potential 
benefits  of  developing  the  system  for  operational  Air  Force  use.  The  test  provides  strong 
evidence  that  IMIS  can  enhance  the  performance  of  maintenance  technicians  performing  on-A/C 
maintenance. 

Technicians  were  able  to  complete  fault  isolation  and  repair  problems  with  greater  accuracy, 
in  a  shorter  time,  and  with  fewer  errors  and  fewer  parts  used  in  the  process.  The  improved 
performance  will  significantly  reduce  the  time  required  to  return  an  A/C  to  operationally  capable 
status,  provide  for  more  effective  utilization  of  available  personnel,  and  reduce  expenditures  for 
procuring,  repairing,  and  stocking  A/C  replacement  components. 

The  test  also  showed  the  advantages  of  the  IMIS  RF  link.  During  the  Fault  Isolation  Test, 
the  RF  was  used  for  ordering  parts  and  transmitting  work  order  close-out  information  which  was 
the  major  contributor  to  faster  job  completion.  The  End-to-End  Demonstration  showed  the 
advantages  of  the  IMIS  RF  capability  in  supporting  flightline  maintenance  managers  in  a  variety 
of  ways,  including  providing  up-to-the-minute  A/C  and  maintenance  status  information  and 
offering  the  capability  to  make  personnel  assignments,  open  and  close  work  orders,  and 
communicate  with  other  maintenance  managers. 

The  Fault  Isolation  Test  demonstrated  that  IMIS  is  easy  to  use  and  is  preferred  by 
technicians.  The  technicians  had  very  little  trouble  using  IMIS  after  a  short  training  session. 
Both  the  specialist  and  APG  technicians  were  able  to  perform  the  test  tasks  with  minimal 
difficulty.  In  contrast,  the  APG  technicians  (and  some  specialists)  experienced  significant 
difficulties  in  using  the  paper  TOs. 

More  detailed  Fault  Isolation  Test  conclusions  are  documented  in  separate  Armstrong 
Laboratory  reports  (AL/HR-TP- 1995-0033  and  AL/HR-TP- 1995-0034). 


LESSONS  LEARNED  AND  RECOMMENDATIONS 

The  IMIS  field  tests  and  demonstrations  proved  to  be  a  valuable  source  of  information 
regarding  the  IMIS  concept  and  implementation.  In  addition,  the  experiences  over  the  entire  life 
of  the  program  provided  information  in  many  different  areas.  The  lessons  learned  and  resulting 
recommendations  regarding  the  hardware,  software,  system,  functionality,  human  factors,  data, 
and  programmatics  are  described  in  the  following  subsections. 
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Hardware 


The  lessons  learned  and  recommendations  regarding  the  PMA,  Maintenance  Information 
Workstation  (MIW),  and  Aircraft  Interface  Panel  (AIP)  are  summarized  in  the  following 
subsections. 

Portable  Maintenance  Aid  (PMA) 

The  overall  weight  and  dimensions  of  the  PMA  were  found  to  be  acceptable  by  the  users. 
There  were  no  complaints  about  the  device  being  too  heavy.  Although  the  length  of  the  PMA 
was  within  specification  when  used  in  the  cockpit  during  the  Fault  Isolation  Test  with  the  handle 
fiilly  extended,  it  occasionally  obstructed  several  instruments.  Possible  reductions  in  the  length 
of  the  PMA  should  be  investigated. 

The  PMA  handle  hinges  experienced  some  durability  problems.  The  oscilloscope-type 
ratchet  joints  were  not  designed  to  withstand  substantial  weights  and  were  not  force- 
synchronized  (i.e.,  it  was  possible  to  turn  one  and  not  the  other).  Additional  reinforcement 
should  be  provided  to  make  the  handles  sturdier.  In  addition,  the  movement  of  the  handle  was 
restricted  somewhat  by  the  RF  antenna  and  the  connectors.  If  the  antenna  were  hinged,  the 
handle  could  be  fully  extended  without  interference.  This  would  also  allow  the  antenna  to  be 
adjusted  to  enhance  reception.  Other  locations  for  the  connectors  should  be  considered  as  well. 

The  PMA  strap  was  used  only  sporadieally  during  the  field  test  because  additional 
persormel  were  available  to  assist  the  technician  in  taking  required  equipment  to  the  A/C.  The 
use  of  the  strap  should  be  evaluated  further  to  determine  its  usefulness. 

The  tethers  for  the  protective  caps  for  the  PMA  connectors  (external  power,  keyboard, 
1553,  etc.)  broke  easily.  A  sturdier  mechanism  should  be  used  to  attach  the  caps  to  the 
connectors. 

There  are  two  1553  interface  cables  for  the  PMA;  each  cable  has  the  same  connector. 
Using  the  same  connectors  allowed  the  technician  to  connected  the  cables  into  the  wrong  port 
resulting  in  misidentification  of  the  discrepancy.  With  the  small  number  of  pins  actually  used,  it 
would  be  feasible  to  combine  the  two  channels  in  a  single  connector  to  eliminate  confusion. 

A  close-fitting  right-angle  connector  (or  another  type  of  connector  which  does  not  stick 
straight  out  from  the  case)  should  be  considered  for  the  external  power  connector.  This  would 
reduce  the  likelihood  of  damage  to  the  connector  or  the  PMA  if  bumped  and  would  reduce  the 
space  occupied  by  a  plugged-in  PMA.  This  was  especially  a  problem  when  multiple  PMAs  were 
being  charged  at  one  time.  A  rack  in  the  support  area  with  which  several  PMAs  could  be  docked 
to  charge  their  batteries  could  be  developed. 

The  PMA  power  switch  is  a  two-position,  standard  toggle  switch  which  is  not  entirely 
recessed  and  has  the  potential  for  being  inadvertently  turned  off.  In  a  UNIX  environment,  where 
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improper  shutdown  of  the  PMA  may  corrupt  the  file  system,  a  totally  recessed  locking  toggle 
switch  or  guarded  toggle  switch  should  be  considered. 

Navigation  using  the  thumb  knob  on  the  PMA  was  not  viewed  favorably  by  users.  The 
select  function  was  sometimes  inadvertently  activated  when  the  user  intended  to  move  the  cursor. 
A  pen-based  or  touch-screen  user  interface  (or  possibly  other  pointing  devices  which  have 
become  available)  should  be  considered,  especially  for  the  flightline  expediters'  devices.  This 
would  be  particularly  useful  on  the  PMA's  virtual  keyboard,  rather  than  the  awkward  navigation 
and  selection  using  the  arrow  and  function  keys. 

The  3/4"  keys  and  1/4"  spacing  required  by  MIL-STD-1472  limited  the  number  of  keys 
which  could  be  on  the  PMA  and  may  have  an  impact  on  future  PMA  dimensions.  Tailoring  of 
MIL-STD-1472  requirements  to  allow  smaller  keys  and  narrower  spacing  could  provide  space 
for  additional  keys  or  allow  the  dimensions  of  the  PMA  to  be  reduced  if  no  additional  keys  are 
necessary.  In  addition,  it  was  sometimes  difficult  to  tell  whether  the  bubble  dome  type  keys  used 
on  the  PMA  keypad  had  responded  to  a  key  press;  thus  the  user  would  press  the  key  numerous 
times.  The  tactile  feedback  provided  by  the  keypad  did  not  guarantee  that  electrical  contact  had 
been  made.  Lowering  the  resistance  in  the  keypad,  thereby  reducing  the  force  required  to  press 
the  PMA  key,  could  also  enhance  key  activation. 

The  battery  life  was  between  2.5  and  4  hours,  which  was  adequate  for  most  diagnostics 
tasks.  When  batteries  were  not  allowed  to  discharge  fully  before  recharging,  "memory  effect" 
problems  were  experienced  due  to  the  properties  of  the  nickel-cadmium  battery  pack.  Periodic 
reconditioning  restored  many  of  the  batteries  to  a  useful  charge  life.  The  lack  of  adequate  time 
between  the  time  a  low-battery  warning  appears  and  the  time  failure  occurs,  another 
characteristic  of  the  nickel-cadmium  batteries  used,  might  dictate  the  use  of  a  battery  having  a 
more  gradual  discharge  rate  at  the  end  of  its  cycle.  Other  rechargeable  battery  technologies  (such 
as  nickel-metal  hydride  or  lithium)  should  be  evaluated. 

PMA  battery  replacement  was  awkward.  The  captive  screws  which  held  the  cover  in 
place  were  very  small,  and  the  cover  was  not  hinged  or  otherwise  fastened  to  the  body  of  the 
PMA.  Some  non-volatile  memory,  which  would  allow  the  PMA  battery  to  be  changed  without 
having  to  shut  it  down  all  the  way,  would  be  desirable. 

High  temperatures  and  direct  sunlight  affected  the  readability  of  the  PMA  display,  which 
would  become  darker.  The  adjustment  of  contrast  via  software  accessible  through  the  main 
menu  bar  is  not  adequate  if  screen  visibility  has  already  deteriorated  to  the  point  where  the  screen 
is  unreadable.  The  contrast  adjustment  should  be  on  the  PMA  box  itself  In  addition,  glare  and 
off-angle  visibility  need  to  be  improved.  Displays  considered  for  future  PMAs  should  undergo 
thorough  evaluations  and  tradeoffs  considering  these  factors  as  well  as  cost  and  power. 

The  PMA  would  have  benefited  from  a  more  modular  design  regarding  maintainability, 
upgradeability,  and  testability.  A  production  PMA  should  include  test  points  or  ports  and  a 
modular  design  such  that  each  of  the  major  components  could  be  removed  and  replaced 
independently  for  maintenance  and  upgrades. 
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Maintenance  Information  Workstation  (MIW)  and  Memory  Module  Loader  (MML) 

The  MIW  and  MML  were  viewed  very  favorably,  with  minimal  problems  experienced. 
Use  of  commercial  off-the-shelf  (COTS)  hardware  was  the  most  efficient  solution  for  these 
hardware  devices. 

Aircraft  Interface  Panel  (AIP) 

The  AIP  implementation  can  range  from  an  on-board  PMA-like  device  to  simply 
providing  surface  access  to  the  data  bus  controller.  For  the  F-16,  it  would  be  adequate  to  provide 
access  to  the  1553  bus  controller  via  a  surface  attachment.  If  maintenance  is  done  in  an  isolated 
environment,  it  may  be  better  to  have  an  on-board  PMA-like  device  which  can  be  used  to  display 
TOs,  Job  Guides,  checklists,  and  so  forth,  thereby  eliminating  the  need  to  have  a  PMA  available. 

Software 

The  screen-based  CAMS  interface  was  inefficient  and  susceptible  to  frequent 
interruptions  due  to  minor  software  changes.  A  standard  legacy  database  interface,  which  might 
require  changes  to  both  CAMS  and  IMIS,  should  be  developed.  A  better  method  would  utilize 
Structured  Query  Language  (SQL). 

Information  regarding  updates  to  CAMS  was  not  disseminated  in  a  timely  manner  to  the 
organizations  which  needed  access  to  that  information.  The  format  in  which  this  information 
was  received  also  resulted  in  additional  time  and  effort  spent  communicating  the  changes  to  the 
subcontractor.  In  addition,  limited  information  was  available  to  assist  in  understanding  all  the 
data  elements  and  values  used  in  CAMS.  For  a  fully  implemented  CAMS  interface,  the 
development  of  a  CAMS  data  dictionary,  in  which  all  attributes,  values,  and  error  messages  are 
defined  in  detail,  would  be  required. 

Modem  connection  should  be  available  between  all  software  development  and  test  sites. 
The  connection  provided  a  very  efficient  mechanism  for  remotely  troubleshooting  software  and 
testing  the  CAMS  interface  without  being  at  Luke  AFB. 

The  demonstration  system  was  saturated  by  the  RF  messages  sent  to  a  relatively  small 
number  of  PMAs.  To  support  messaging  in  an  unconstrained  environment  when  dealing  with  the 
number  of  PMAs  required  to  support  an  entire  FS,  a  mechanism  for  filtering  messages  may  be 
required.  Improving  the  efficiency  of  parsing  the  messages  into  the  database  would  also  help 
alleviate  this  problem. 

The  tools  that  were  selected  for  the  software  development  environment  were  not 
sufficiently  mature  to  support  the  functional  and  performance  requirements.  Throughout 
software  development,  various  bugs  and  limitations  were  encountered,  with  very  few  quick  fixes. 
A  more  extensive  analysis  of  the  available  tools  and  platforms  should  be  conducted  before 
beginning  implementation. 
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The  Santa  Cruz  Operation  (SCO)  UNIX,  selected  as  the  PMA  operating  system,  was  not 
as  standard  as  originally  believed.  Upgrades  to  other  tools  could  not  be  made  without 
undergoing  an  additional  process  to  re-host  under  SCO  UNIX  (often  at  additional  cost). 
Choosing  a  second  operating  system  also  resulted  in  additional  effort  to  recompile  source  code 
and  to  test  and  fix  platform-specific  problems. 

The  design  used  for  this  demonstration  implementation  was  not  developed  in  as  modular 
a  fashion  as  would  be  expected  for  a  production  system.  Direct  calls  to  the  database  and  the 
Graphical  User  Interface  (GUI)  tool  are  found  throughout  the  application  code,  so  that  when  a 
change  to  a  specific  database  or  GUI  call  was  needed,  a  time-consuming  search  of  the  code  for 
these  calls  was  required. 

Field  experience  with  the  PMAs  has  generated  concern  over  performance  of  the  current 
80X86-based  architecture,  in  spite  of  two  processor  upgrades.  Performance  requirements  should 
be  determined  early  in  the  design  process  so  that  the  appropriate  processor  and  system 
architecture  can  be  chosen.  System  performance,  although  not  always  a  requirement,  is  always 
an  issue. 

Aural  and  more  attention-getting  visual  alerts  regarding  incoming  messages  should  be 
available  on  both  the  PMA  and  the  MIW,  especially  for  time-critical  messages  for  certain  users 
like  the  expediters  or  the  production  superintendent.  The  display  of  the  message  icon  did  not 
provide  adequate  notice  to  the  user  that  a  message  had  been  received. 

Separate  databases  containing  the  diagnostics  and  TO  data  for  each  subsystem  were 
inefficient.  The  technician  was  able  to  perform  diagnostics  tasks  for  only  a  single  subsystem.  If 
the  technician  needed  to  perform  another  diagnostics  task  on  a  different  subsystem,  they  had  to 
return  to  the  support  section  to  check  out  a  new  cartridge.  This  also  required  that  multiple 
databases  on  the  MML  be  updated  and  maintained.  It  would  be  better  to  provide  all  diagnostics 
and  TO  data  on  each  cartridge  or  to  provide  a  broader  set  of  subsystems  to  be  supported  by  a 
single  cartridge. 

The  1553  interface,  as  implemented,  was  inefficient  in  that  every  new  communication 
had  to  be  developed  and  tested  for  each  subsystem.  The  preferred  solution  would  be  to  have  the 
PMA  recognized  as  a  remote  terminal  device  by  the  bus  controller's  Operational  Flight  Program 
(OFP). 


The  "Back"  capability  (ability  to  back  up  to  a  previous  step  or  screen)  should  be 
implemented  throughout,  both  in  IMIS  and  in  TO  Present.  However,  the  backward  navigation 
implemented  in  TO  Present  displayed  messages  pertaining  to  software  states  which  were  not 
relevant  to  the  user.  These  messages  should  be  removed  or  replaced  with  messages  that  are 
pertinent  to  the  back-up  sequence. 

The  TO  Presentation  system  should  allow  the  user  to  view  more  than  just  the  current  step. 
The  display  should  include  prior  and  upcoming  steps,  with  the  current  step  highlighted. 
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Actuation  of  the  “F8  Next”  function  key  would  then  move  the  highlighted  area  to  the  next  step. 
A  "review"  capability,  which  would  allow  the  user  to  review  any  step  in  the  entire  task  without 
changing  the  state  table  values,  should  also  be  considered. 

When  displaying  TO  input  conditions,  such  as  number  of  people,  support  equipment,  and 
consumables  required,  these  conditions  should  be  grouped  on  a  screen.  Placing  each  of  these 
items  on  a  separate  screen  reduces  the  technician's  efficiency. 

Utilities  to  perform  certain  database  changes  (such  as  updating  the  list  of  A/C  or 
personnel)  would  have  provided  a  much  faster  mechanism  for  updating  databases.  With  the 
current  software,  many  of  these  changes  could  only  be  done  by  building  a  new  database,  a 
process  which  takes  a  significant  number  of  hours  between  creation  and  testing. 

More  automation  of  the  software  installation  process  would  have  been  beneficial.  With 
the  frequent  field  updates,  there  were  often  problems  with  having  the  correct  versions  of  all  the 
files  available.  Automating  this  process  would  have  eliminated  many  of  these  errors. 

System 

Wireless  communication  is  required  for  an  effective  maintenance  operational 
environment.  Without  wireless  communication,  flightline  managers  would  not  have  the  most 
current  information  available  to  perform  their  tasks.  The  entire  A/C  maintenance  area  must  be 
supported  by  wireless  communication.  RF  coverage  was  adequate  for  the  flightline  area  but  was 
inconsistent  in  the  hangars,  where  it  was  also  required.  Providing  large  area  coverage  may 
require  the  use  of  repeaters. 

Security  issues  are  a  potential  problem  for  a  system  which  has  access  to  classified  data. 
Use  of  classified  TOs  on  the  PMA,  when  a  user  does  not  always  maintain  constant  possession  of 
the  PMA,  needs  to  be  addressed.  This  will  also  be  a  problem  for  displaying  classified  data  on  the 
MIW  in  areas  that  are  not  adequately  controlled. 

Functionality 

The  lessons  learned  and  recommendations  regarding  the  functionality  in  each  IMIS 
functional  area  are  described  in  the  following  subsections. 

Startup/ Shutdown 

The  text  shown  by  the  operating  system  during  startup  and  shutdown  should  be  hidden 
from  normal  PMA  users.  It  should  be  available  on  request  to  system  admini.strators  and  software 
engineers.  In  addition,  the  user  should  not  have  to  respond  to  any  prompts  when  starting  up  or 
shutting  down  the  PMA.  The  RF  should  always  be  turned  on;  the  user  can  turn  off  the  RF,  if 
desired,  after  logging  in.  This  would  make  the  startup  and  shutdown  procedures  less  confusing 
for  the  user.  Also,  the  software  control  of  the  backlight  intensity  when  bringing  up  and  shutting 
down  the  PMA  was  limiting.  The  user  could  not  always  see  the  displayed  text. 
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The  users  also  expressed  a  desire  to  have  an  indication  of  whether  their  assigned  PMA  is 
fully  functional.  The  results  of  the  PMA's  power-on  Built-In-Test  (BIT)  could  be  reported  to  the 
user,  just  prior  to  the  IMIS  Login  screen,  to  provide  this  information. 

Login 


The  users  noted  that  they  normally  access  the  same  information  each  time  they  log  onto 
IMIS.  Displaying  the  most  frequently  used  screen  for  that  particular  duty  title  immediately  after 
logging  on  could  be  more  efficient  than  requiring  the  user  to  select  the  menu  item.  For  example, 
airplane  general  (APG)  flightline  expediters  would  want  to  see  the  FS  A/C  Status,  while 
maintenance  debriefers  would  want  the  list  of  undebriefed  sorties  shown  prior  to  performing 
debrief 


The  "Welcome  to  IMIS"  screen  following  Login  which  provides  the  name  as  "Smith  John 
Z"  should  be  changed  to  include  the  user's  rank  and  last  name  only  (e.g.,  Sgt.  Smith). 

Menu  Bar 

When  in  TO  Present,  menu  accelerators  (i.e.,  using  numbers  to  select  menu  items)  were 
available.  The  users  found  this  to  be  a  desirable  feature  which  they  recommended  should  be 
available  throughout  IMIS,  not  just  in  TO  Present.  This  could  also  be  expanded  for  application 
to  data  listers. 

Prepare/Extract  PMA  Cartridge 

Creation  of  a  PMA  cartridge  should  be  based  on  a  specific  technician  assigned  to  work  on 
a  specific  discrepancy,  not  on  a  subsystem  list.  This  could  be  accomplished  by  providing  to  the 
support  personnel  at  the  start  of  Prepare  PMA  Cartridge  a  list  of  open  work  orders  (to  include  the 
assigned  technician  and  scheduled  start  time)  for  which  a  cartridge  has  not  yet  been  prepared. 
The  work  unit  code  (WUC)  of  the  work  order  would  then  be  used  to  identify  the  corresponding 
subsystem  to  be  worked.  This  process  would  also  assist  in  identifying  who  has  been  assigned  to 
each  PMA,  a  manual  process  in  the  current  system. 

The  Extract  PMA  Cartridge  Data  function  should  include  retrieving  audit  trail  data 
recorded  during  a  diagnostics  session  and  storing  it  on  the  MIW  for  further  analysis. 

Task  Assignment 

There  were  several  aspects  of  the  Task  Assignment  display  in  terms  of  usability  and 
readability  which  the  users  found  could  be  improved.  For  example,  deletion  of  a  user's  final  task 
assignment  should  not  delete  the  user's  name  from  the  display,  and  repetition  of  a  person's  name 
when  that  person  is  assigned  multiple  tasks  makes  the  display  unnecessarily  difficult  to  read. 
They  also  suggested  that  the  display  include  the  time  the  assignment  was  made  and  the  time  the 
work  was  started  (for  work  orders  only). 
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When  assigning  tasks  and  requesting  the  list  of  job  control  numbers  (JCNs),  the 
unassigned  JCNs  for  that  manager  should  be  displayed  first  on  the  list.  Currently,  all  open  JCNs 
are  displayed,  resulting  in  a  very  lengthy  list.  Reading  this  list  would  also  be  easier  if  the  A/C 
tail  number  and  WUC  were  provided  along  with  the  JCN. 

If  the  person  for  whom  a  task  assignment  change  has  been  made  is  currently  logged  in, 
that  person  should  be  notified  of  the  change  via  a  message.  This  will  help  users  keep  informed 
of  task  assignment  changes  that  affect  them. 

The  capability  to  display  a  list  of  open  work  orders  assigned  to  a  particular  work  center  or 
aircraft  based  on  the  WUC  should  be  provided.  This  information  would  allow  a  manager  to 
know  what  other  tasks  are  currently  in  work  and  whether  the  current  assignments  should  be 
reassigned  to  accommodate  an  incoming  work  order. 

Certification  roster  information  should  be  readily  available  (i.e.,  by  pressing  a  single 
function  key)  when  making  task  assignments  to  check  personnel  qualifications  against  the 
specific  task.  This  would  be  beneficial  information  to  have  available  when  deciding  which 
individual  should  be  assigned  to  a  particular  task. 

Expediters  who  update  the  task  assignments  should  be  able  to  see  and  update  only  the 
personnel  in  their  work  center  and  on  their  shift.  The  production  superintendent  and  flight  chiefs 
should  have  the  ability  to  update  and  review  all  personnel  assignments  for  their  work  center. 

Certification  Roster 

The  certification  roster  is  currently  a  read-only  display.  In  an  operational  environment,  it 
should  be  possible  for  the  authorized  personnel  to  make  certification  roster  updates  in  IMIS  and 
transmit  the  changes  to  the  appropriate  legacy  systems. 

In  addition  to  displaying  the  certification  roster  by  person,  the  capability  to  select  a 
qualification  and  list  all  personnel  who  have  that  particular  qualification  should  be  provided. 
This  would  be  useful  in  making  task  assignments.  Certain  certification  data  (e.g.,  clearing  a  red 
X,  back  ordering  a  part,  or  performing  an  engine  run)  may  need  to  be  available  during  the  task. 
This  data  could  be  inserted  into  the  technical  data  or  have  a  soft  key  to  bring  up  the  certification 
roster. 

Aircraft  Status 

The  APG  expediters,  specialist  expediter,  production  superintendent,  and  munitions 
expediter  will  all  want  to  record  remarks  on  the  status  screen.  The  remarks  field  should  be  made 
editable  for  them  to  record  their  ovm  notes  for  each  A/C.  This  data  may  need  to  be  saved  from 
one  day  to  the  next. 


23 


When  assigning  an  A/C  to  a  location,  the  user  is  not  prevented  from  assigning  it  to  a 
location  already  occupied  by  another  A/C.  Some  locations  allow  only  a  single  A/C,  while  others 
(e.g.,  Wash  Rack)  can  accommodate  multiple  A/C.  IMIS  should  automatically  check  this  data 
for  potential  conflicts  and  alert  the  user  of  any  found. 

The  only  status  attribute  values  that  were  used  were  fully  mission  capable  (FMC), 
partially  mission  capable  (PMC),  and  not  mission  capable  (NMC).  The  rules  for  the  use  of  the 
other  status  codes  which  provide  information  about  the  cause  of  the  status  (i.e.,  supply, 
maintenance,  or  both)  should  be  automated.  An  example  of  this  would  include  updating  status  to 
not  mission  capable  -  supply  (NMCS)  when  a  required  part  is  back  ordered. 

Status  information  is  most  current  in  IMIS,  not  in  a  legacy  database.  IMIS  should 
transmit  A/C  status  updates  to  the  legacy  database  to  ensure  that  the  most  up-to-date  information 
is  available  in  both  systems. 

A/C  estimated  time  in  commission  (ETIC)  should  consider  only  those  work  orders  which 
have  a  grounding  discrepancy  (symbol  of  X).  Currently,  the  ETICs  of  all  work  orders  open 
against  an  A/C  are  used  to  determine  the  overall  A/C  ETIC.  When  the  last  work  order  with  a 
symbol  of  X  is  closed,  the  A/C  ETIC  field  should  be  blank. 

The  specialist  expediter,  munitions  expediter,  and  production  superintendent  need  to  be 
able  to  review  status  for  all  A/C  in  the  FS  (not  just  A-Flight  or  B-Flight).  Upon  selection  of  the 
FS  A/C  Status  menu  item,  it  would  also  be  more  efficient  to  display  the  most  frequently  used 
option  rather  than  provide  a  choice.  The  A-Flight  APG  expediter  would  normally  view  the  A- 
Flight  status,  the  B-Flight  APG  expediter  would  normally  view  the  B-Flight  status,  and  the  other 
maintenance  managers  would  review  the  status  of  all  A/C.  The  Different  Organization  function 
key  could  be  used  if  a  user  wants  to  display  data  for  a  different  set  of  A/C. 

Maintenance  Operations  Center  (MOC)  users  require  the  capability  to  view  data  for  all 
squadrons  in  the  wing,  although  much  of  that  data  may  be  preferred  on  a  squadron-by-squadron 
basis.  The  "Different  Organization"  hierarchical  listers  must  include  choices  appropriate  to  users 
throughout  the  wing  and  should  be  tailored  to  the  specific  user. 

When  a  technician  requests  fuel  for  a  particular  A/C,  IMIS  currently  sends  a  message  to 
the  appropriate  APG  expediter,  who  must  then  manually  notify  the  MOC  of  the  fuel  request. 
Upon  acknowledgment  of  the  fuel  request  message  by  the  APG  expediter,  IMIS  should  send  a 
message  to  the  MOC  indicating  which  A/C  needs  fuel  and  the  location  to  which  the  fuel  truck 
should  be  sent. 

The  use  of  the  mail  icon  for  processing  fuel  requests,  exceptional  release  requests,  and 
status  change  requests  may  become  a  burden  for  the  APG  expediter.  The  use  of  a  visual 
indicator  on  the  status  screen  (making  the  "Crew  P^eady"  field  flash  when  an  Exceptional  Release 
has  been  requested,  for  example)  might  allow  the  expediter  to  process  the  information  more 
efficiently. 
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The  placement  of  information  on  the  different  duty  title-based  FS  status  screens  was 
made  as  standard  as  possible.  Since  the  user  must  scroll  to  view  all  the  information  available,  it 
would  be  better  to  tailor  each  screen  by  user  group  to  display  the  information  accessed  most 
often  by  each  group.  This  will  reduce  the  amount  of  scrolling  required  to  present  users  with  the 
information  they  use  the  most. 

The  FS  A/C  Status  screen  needs  to  be  read-only  for  the  technician.  The  technician  needs 
only  to  be  allowed  to  recommend  a  change  to  the  ETIC  for  a  single  A/C,  which  should  be  done 
through  the  Individual  A/C  Status  screen. 

The  Configuration  Code  lister  is  not  conducive  to  making  single  changes  to  an  A/C's 
configuration.  To  make  any  change  to  the  configuration  code,  the  user  must  repeat  all  the 
hierarchical  steps,  including  those  which  have  not  changed.  Options  for  making  this  more 
efficient  should  be  investigated. 

There  should  be  a  relationship  between  the  navigation  pod  and  targeting  pod  information 
and  the  configuration  code.  A  change  in  the  navigation  and  targeting  pod  information  should 
cause  an  update  to  the  configuration  code  (and  vice  versa).  The  presence  or  absence  of  these 
pods  can  be  traced  to  the  first  character  of  the  configuration  code. 

The  number  of  days  since  the  last  flight  should  be  automatically  computed  and  updated 
based  on  each  day's  actual  sorties  flown,  as  recorded  by  IMIS.  It  Will  be  very  time-consuming 
for  the  user  to  determine  this  information  manually. 

The  MOC's  sortie  summary  (available  through  the  FS  A/C  Status  screen)  should  reflect 
the  flying  schedule  and  debrief  information  available  each  day.  The  totals  should  be  reset  as 
appropriate  at  the  start  of  each  day. 

ETIC  and  status  updates  for  all  A/C  are  sent  to  the  production  superintendent  and  the 
appropriate  APG  expediter.  These  updates  should  also  be  sent  to  the  specialist  expediter  and  the 
munitions  expediter. 

Debrief 

After  completing  a  debrief,  the  user  should  return  to  the  list  of  undebriefed  sorties,  not  the 
main  menu  bar.  The  maintenance  debriefers  will  normally  conduct  several  debrief  sessions  in  a 
row,  and  this  change  would  allow  the  debriefer  to  begin  the  next  debrief  more  quickly.  Once  all 
sorties  had  been  debriefed,  a  message  to  that  effect  would  be  displayed  prior  to  returning  the  user 
to  the  main  menu  b^lr. 

There  is  currently  no  capability  to  edit  debrief  information  within  IMIS  after  the 
maintenance  debriefer  finishes  the  debrief  and  prints  the  sortie  recap  (which  also  causes  the 
debrief  data  to  be  sent  to  the  legacy  system).  If  incorrect  information  is  provided  (or  if  the 
information  changes  following  debrief),  the  user  must  currently  use  CAMS  to  edit  the  debrief 
data.  IMIS  should  provide  access  to  a  specified  amount  of  debrief  data  (one  or  two  days. 
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possibly)  for  editing  and  re-transmission  to  CAMS.  This  could  also  provide  the  capability  to 
reprint  the  sortie  recap  in  the  event  there  was  a  printer  failure  when  the  recap  was  originally  sent 
to  the  printer. 

No  capability  was  provided  to  debrief  an  unscheduled  sortie.  This  option  should  be 
provided  to  allow  debriefing  of  unscheduled  sorties  and  sorties  for  which  flying  schedule 
changes  have  not  yet  been  entered. 

The  users  requested  that  the  Julian  date  and  local  time  be  displayed  at  all  times.  This 
would  be  especially  valuable  during  debrief. 

The  scheduled  take-off  and  landing  times  were  not  included  in  the  debrief  display. 
Consequently,  debriefers  sometimes  canceled  out  of  debrief  transactions  to  ensure  that  the  data 
entered  would  not  result  in  a  sortie  deviation. 

When  opening  a  work  order  during  debrief,  the  ETIC  and  scheduled  start  date  and  time 
should  be  left  blank  (or  deleted  entirely  from  the  display).  The  maintenance  debriefer  does  not 
have  the  necessary  information  to  assess  when  the  task  will  start  and  how  long  it  will  take  to 
complete.  The  expediter  will  be  able  to  make  this  determination  when  the  task  is  assigned. 

Pilot  Call-In 

Depending  on  the  base,  the  pilot  call-in  information  may  actually  be  received  by  the  FS 
Operations  Center  personnel  or  by  the  MOC.  An  MIW  needs  to  be  available  in  the  FS 
Operations  Center  to  support  entry  of  pilot  call-in  data. 

Minimum  Essential  Subsystem  List 

The  Minimum  Essential  Subsystem  List  (MESL)  is  currently  a  read-only  display.  When 
the  MESL  needs  to  be  updated,  password-controlled  changes  by  authorized  users  should  be 
allowed. 

Work  Order 

The  status  changes  which  are  recommended  by  IMIS  upon  the  opening  or  closing  of  a 
work  order  should  take  the  MESL  and  the  next  scheduled  mission  for  the  A/C  into  account. 
Currently,  the  status  is  based  strictly  on  the  symbol  for  the  work  order  (X  causes  the  A/C  to  be 
NMC,  while  /  causes  it  to  be  PMC). 

The  WUC  should  be  displayed  more  prominently  on  the  Open  Work  Order  screen.  The 
WUC  can  also  be  used  to  determine  which  work  center  the  task  should  be  assigned.  Once  the 
fault  code  is  known,  this  may  provide  information  driving  a  change  to  the  ETIC. 

The  When  Discovered  Code  (WDC)  should  be  validated  against  the  debrief  data  (if 
opened  during  debrief)  and  the  flying  schedule.  For  example,  if  the  WDC  indicates  that  the 
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discrepancy  caused  an  air  abort,  the  debrief  for  that  sortie  should  reflect  the  same  information. 
Also,  if  the  discrepancy  is  discovered  several  hours  before  the  scheduled  sortie,  the  WDC  should 
be  consistent  with  this  fact. 

The  assignment  of  a  work  order  to  a  technician  should  be  based  on  both  qualification  and 
availability.  When  the  expediter  receives  the  open  work  order  notification,  it  would  be  helpful  to 
have  access  to  a  list  of  technicians  within  that  expediter's  organization  who  are  qualified,  with  an 
indication  of  whether  each  is  currently  available.  If  there  are  no  qualified  technicians  available, 
the  expediter  would  have  the  option  to  schedule  the  work  order  for  a  later  time  when  the 
technician  is  available  or  to  select  a  technician  who  is  currently  assigned  to  a  different  task  and 
juggle  the  task  assignment  schedule. 

The  opening  of  certain  panels  requires  that  additional  work  orders  be  generated.  IMIS 
could  increase  the  efficiency  of  the  maintenance  process  if  these  work  orders  could  be  opened 
automatically  when  the  corresponding  step  in  a  TO  is  accessed  or  completed. 

The  Performing  Work  Center  (PWC)  field  should  not  be  editable  on  the  Update  Work 
Order  screen.  If  the  PWC  needs  to  be  changed,  the  technician  should  generate  a  Work  Center 
Event  (WCE),  not  update  the  work  order. 

The  Work  Order  History  display  should  also  include  the  Action  Taken  code.  How 
Malfunctioned  code,  symbol,  WDC,  the  person  who  closed  the  work  order,  and  the  person  who 
inspected  the  work. 

The  Work  Order  History  displayed  should  be  based  on  a  match  of  the  entered  characters 
of  the  WUC,  not  all  five  characters.  When  the  work  order  is  opened,  only  the  first  three 
characters  are  typically  known  (e.g.,  74A00),  while  the  closed  work  order  will  have  more  detail 
in  the  WUC.  A  match  of  the  open  work  order  WUC  to  closed  work  order  history  WUCs  can 
occur  only  if  the  number  of  WUC  characters  compared  is  the  same  for  the  closed  work  orders  as 
is  known  for  the  open  work  order  or  specified  in  the  query. 

The  closed  work  order  notification  should  also  be  sent  to  the  person  in  charge  of  the 
organization  to  which  the  work  order  was  assigned  (e.g.,  the  specialist  expediter  should  be 
notified  when  a  work  order  assigned  to  the  Avionics  Section  is  closed).  This  will  provide  the 
expediter  with  information  about  the  availability  of  personnel. 

Information  regarding  removed  and  installed  part  and  serial  numbers  obtained  during 
diagnostics  should  be  automatically  pre-filled  for  the  closed  work  order.  Ideally,  this 
information  would  not  even  have  to  be  entered  during  diagnostics  (the  information  for  the 
removed  part  could  be  determined  from  the  A/C,  while  the  information  for  the  installed  part 
could  be  provided  by  supply). 

Red  X  assist  messages  should  not  always  be  sent  to  the  APG  expediter.  When  the  WUC 
of  the  work  order  is  for  an  avionics  subsystem,  the  Red  X  assist  message  should  be  sent  to  the 
specialist  expediter.  Additionally,  when  the  APG  or  specialist  expediter  receives  the  Red  X 
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assist  message,  he  or  she  should  be  able  to  display  a  list  of  personnel  who  have  the  necessary 
qualifications,  select  a  name  from  the  list,  and  send  a  message  to  that  person,  indicating  the  A/C 
tail  number  and  location  to  which  that  person  should  report. 

Entering  multiple  criteria  (A/C  tail  number,  performing  work  center,  or  JCN)  for 
identifying  a  work  order  to  be  closed  or  updated  resulted  in  a  dramatic  increase  in  the  amount  of 
processing  time.  The  method  used  to  search  the  database  for  work  orders  meeting  these  criteria 
should  be  examined  to  determine  how  this  can  be  made  more  efficient. 

Schedules 

The  specialist  expediter,  munitions  expediter,  and  production  superintendent  need  to  be 
able  to  review  the  flying  and  maintenance  schedules  for  all  A/C  in  the  FS  (not  just  A-Flight  or  B- 
Flight).  Upon  selection  of  any  of  the  schedule  menu  items,  it  would  be  more  efficient  to  display 
the  most  frequently  used  option  rather  than  provide  a  choice.  It  was  believed  that  the  A-Flight 
APG  expediter  would  normally  view  the  A-Flight  schedules,  the  B-Flight  APG  expediter  would 
normally  view  the  B-Flight  schedules,  and  the  other  maintenance  managers  would  review  the 
schedules  of  all  A/C.  The  Different  Organization  function  key  could  be  used  if  a  user  wanted  to 
display  data  for  a  different  set  of  A/C. 

Adding  a  tail  number  to  the  flying  schedule  involves  coordination  among  the  production 
superintendent,  MOC  senior  controller,  the  FS  Officer-in-Charge  (OIC),  operations  programmers 
and  top-level  management  personnel,  weapons  storage  and  weapons  flight  personnel,  and  the 
Squadron  Plans  and  Scheduling  office.  These  personnel  are  required  to  sign  an  Air  Force  Form 
2407.  IMIS  should  assist  in  coordinating  the  flying  schedule  change  and  obtaining  approval 
from  the  necessary  parties  before  updating  the  database  to  reflect  the  change. 

IMIS  should  have  the  capability  to  create  and  update  schedules  and  send  this  information 
to  the  legacy  database.  Currently,  IMIS  can  only  receive  this  information  from  CAMS,  even 
though  IMIS  may  have  more  up-to-date  information. 

The  list  of  events  displayed  on  the  Maintenance  Schedule  should  be  dynamically 
determined  from  the  maintenance  requirements  found  in  the  time  distribution  listing  and  on  Air 
Force  Technical  Order  (AFTO)  Form  78 IK.  IMIS  currently  displays  a  fixed  set  of  event  types 
which  appear  regardless  of  whether  any  A/C  has  been  scheduled  for  that  activity.  More 
automation  should  also  be  introduced  in  scheduling  A/C  for  periodic,  regularly  scheduled 
maintenance  actions. 

Troubleshooting 

The  Troubleshooting  Started  message  sent  at  the  beginning  of  the  diagnostics  task  should 
include  the  time  started  and  the  assigned  technician,  in  addition  to  the  A/C  tail  number  and  JCN. 
Without  this  additional  data,  the  expediter  receiving  the  message  will  not  have  enough 
information  about  what  is  being  done  to  each  A/C. 
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The  ability  to  review  parts  status  and  to  review  messages  while  in  TO  Present  is  essential. 
Although  messages  can  be  received  and  displayed  while  in  TO  Present,  the  message  icon  does 
not  appear,  so  the  user  is  not  made  aware  that  a  message  (often  a  part  status  message)  has  been 
received. 

The  existing  bookmark  capability  cannot  be  used  to  annotate  problems  with  the  TO  data 
for  use  in  filling  out  an  AFTO  Form  22,  since  the  bookmark  is  not  saved  after  the  diagnostics 
session  is  closed.  The  capability  to  save  such  a  bookmark  to  allow  the  AFTO  Form  22  to  be 
generated  at  a  later  time,  on  the  MIW,  would  be  desirable. 

The  diagnostics  functionality  provided  (change  test  results,  list  of  actions  taken,  rankings 
of  tests/repairs/actions,  ete.)  was  valuable  to  technicians  and  helped  make  their  job  more 
efficient.  In  addition,  it  was  useful  to  display  the  recommended  best  action  prominently  and 
allow  that  procedure  to  be  initiated  via  a  single  keystroke,  since  this  was  the  option  most 
frequently  selected. 

Teehnieians  should  not  be  given  the  option  to  work  on  JCNs  that  are  not  assigned  to 
them.  Currently,  users  are  notified  at  the  beginning  of  troubleshooting  when  they  have  no  tasks 
assigned  to  them  and  are  given  the  option  to  seleet  from  other  work  orders.  This  capability 
should  be  eliminated. 

The  dialog  box  which  asked  the  technician  if  enhanced  diagnostics  should  be  used  was 
confusing  for  most,  in  that  they  were  told  to  respond  with  "F2  No."  This  dialog  box  should  be 
reworded  to  define  more  clearly  the  consequences  of  each  response,  and  the  response  which  will 
be  seleeted  most  frequently  should  correspond  to  FI. 

Parts 


Having  part  availability  information  when  ordering  a  part  through  diagnostics  was 
valuable.  This  information  should  also  be  provided  when  ordering  a  part  outside  of  diagnostics. 
At  a  minimum,  the  software  should  eheek  to  see  if  the  latest  inventory  levels  from  the  Standard 
Base  Supply  System  (SBSS)  show  whether  that  part  is  in  stock.  If  there  appears  to  be  none  in 
stock,  the  technician  should  receive  a  message  stating  this  and  should  be  given  options  regarding 
how  to  proceed  (e.g.,  call  production  superintendent). 

Although  the  Quick  Reference  List  (QRL)  is  not  used  at  all  bases,  the  users  at  Luke  AFB 
liked  having  the  electronic  version  of  the  QRL  and  found  this  to  be  an  efficient  mechanism  for 
ordering  the  most  frequently  needed  parts. 

Since  all  part  orders  are  authorized  by  the  production  superintendent  (and  supply 
personnel  do  not  assist  in  filling  out  the  appropriate  forms),  additional  information  regarding  the 
different  data  elements  on  the  part  order  screen  (Urgeney  Justification  code.  Transaction 
Exception  code.  Mission  Capable  eode,  priority,  etc.)  and  their  values  must  be  available  to  allow 
the  production  superintendent  to  provide  eorrect  input  on  the  part  order,  not  just  to  enter  his  or 
her  password. 
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The  Illustrated  Parts  Breakdown  (IPB)  should  begin  with  a  subsystem  graphic  or  series  of 
graphics  and  use  hot  spots  to  allow  various  components  to  be  selected.  Upon  selecting  a 
component  which  can  be  ordered,  the  textual  part  information  associated  with  that  part  would  be 
displayed.  If  that  component  can  also  be  further  broken  down  into  other  components  which  can 
be  ordered,  these  hot  spots  should  also  be  available  for  selection. 

The  users  emphasized  that  cannibalization  was  very  important  in  the  overall  part-ordering 
process.  This  should  be  one  of  the  areas  considered  for  implementation  in  future  development 
phases. 


The  ability  to  perform  a  defective  part  tum-in  and  generate  the  AFTO  Form  350  tag 
should  be  possible  from  the  flightline.  Currently,  this  capability  is  available  on  the  MIW  only 
(since  printing  is  involved).  Ideally,  this  information  would  then  be  available  when  the  work 
order  is  closed  from  the  flightline,  allowing  the  Tag  Number  field  on  the  Close  Work  Order 
screen  to  be  pre-filled. 

Messages 

When  sending  a  message,  a  list  of  standard  narratives  with  commonly  used  messages 
pertaining  to  that  user's  duty  title  should  be  available  in  addition  to  the  free-form  text  input.  This 
would  save  the  user  considerable  time,  especially  if  a  message  is  being  created  on  a  PMA 
without  a  keyboard. 

The  display  of  messages  one  at  a  time  (and  being  forced  to  select  the  message  icon  or  the 
Display  Messages  menu  bar  option  repeatedly)  was  cumbersome  for  the  user.  The  capability  to 
display  an  index  of  all  unread  messages  (to  include  who  sent  the  message,  date/time  sent,  and 
type  of  message  or  message  subject),  listed  by  priority,  should  be  provided.  This  will  allow  a 
user  who  reeeives  a  lot  of  messages  to  select  the  high  priority  messages  that  need  to  be 
responded  to  most  quickly. 

Notifications  which  are  sent  to  various  personnel  are  based  on  their  duty  title  and  shift, 
regardless  of  whether  they  are  logged  on  or  even  working  that  day.  These  messages  need  to  be 
sent  to  the  people  who  are  available  and  authorized  to  provide  the  neeessary  response.  In 
addition,  if  one  authorized  user  does  not  respond  within  a  specified  amount  of  time,  the  message 
should  be  rerouted  to  another  authorized  user  who  is  available. 

Human  Factors 

There  was  a  high  degree  of  user  acceptance  of  IMIS  despite  the  significant  change  that 
the  system  introduced  from  the  way  the  maintenance  proeess  is  currently  performed.  Early 
involvement  of  the  users  in  both  hardware  and  software  design  was  critical  in  gaining  user 
acceptance. 
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Consistent  placement  of  the  cursor  and  consistent  first  focus  are  necessary  for  the  user  to 
readily  identify  where  the  cursor  control  resides.  There  were  some  cases  when  the  user  did  not 
know  where  data  was  being  entered  or  where  the  cursor  was  even  located  as  a  result.  The  loss  of 
focus  was  a  problem  in  TO  Present  where  the  user  had  to  recognize  that  focus  had  been  lost  then 
move  the  pointer  to  regain  focus. 

Visual  cues  need  to  be  more  apparent  (and  consistent)  when  processing.  The  hourglass 
was  difficult  to  see  and  did  not  always  appear  immediately.  A  dialog  box  might  be  more 
effective  in  certain  cases  where  there  is  a  lengthy  wait.  This  would  help  the  user  avoid  pressing 
keys  inappropriately  when  there  is  some  uncertainty  about  the  status  of  the  processing. 

The  content  of  the  message  line  was  often  inadequate  to  instruct  the  user  of  the  possible 
actions  which  may  be  taken  on  a  given  screen.  In  some  cases,  no  message  was  presented  at  all. 
The  instructions  to  the  user  need  to  be  made  clearer. 

The  screen  format  for  scrollable  displays,  particularly  the  spreadsheets,  made  it  difficult 
for  users  to  know  when  scrolling  would  be  appropriate  or  what  would  be  displayed  if  they  did 
scroll.  It  sometimes  appeared  that  the  entire  spreadsheet  was  already  displayed,  when  in  fact 
there  were  more  rows  or  columns  available.  The  manner  in  which  scrolling  was  performed  on 
the  PMA  was  also  awkward.  It  should  be  noted  that  the  End-to-End  Demonstration  did  not 
provide  the  users  of  most  of  the  spreadsheet  displays  (e.g.,  the  expediters  and  production 
superintendent)  any  extensive  practice  in  using  them  in  a  realistic  context.  This  should  be  further 
evaluated  before  recommending  changes. 


Data 

Skill  levels  (expert  and  novice)  need  to  be  implemented  more  extensively  in  the  data. 
Experienced  users  felt  that  much  of  the  information  that  was  presented  on  the  expert  track  was 
unnecessary. 

The  ability  to  update  data,  change  weights  and  times,  and  update  question  sets  should  be 
provided,  assuming  adequate  controls  can  be  introduced  to  prevent  use  of  unvalidated  data. 
These  controls  must  include  determining  who  the  appropriate  personnel  are  for  updating  this  data 
and  where  these  changes  can  be  made. 

Validation  of  the  TO  data  is  a  significant  issue  because  it  is  very  time-consuming  and 
must  be  repeated  each  time  new  data  is  received  or  data  is  changed.  There  is  currently  no 
mechanism  in  place  for  validating  electronic  TO  data.  Validation  of  the  data  for  use  in  an 
unconstrained  environment  will  be  extremely  costly  for  those  procedures  which  have 
resequenceable  steps  or  which  have  many  branches  to  be  checked.  The  validation  of  data  which 
is  not  resequenceable  (e.g.,  work  cards  or  remove/replace  procedures)  will  not  be  as  difficult. 

Errors  in  identifying  pins  have  always  been  a  local  problem  in  the  maintenance  of 
electronic  equipment.  The  illustrated  connectors  in  the  authored  technical  data  for 
continuity/resistance/voltage  tests  were  so  generic  they  added  little  information  except  perhaps  to 
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remind  the  user  when  measurements  were  to  be  made  between  connectors  on  cables  rather  than 
between  connectors  on  the  line  replaceable  units  (LRUs).  Technical  data  would  have  been  more 
useful  if  blow-ups  (enlargements)  were  made  of  the  connectors  with  callouts  on  the  pins  to  be 
checked.  This  information  to  assist  in  pin  identification  would  be  expensive  to  produce,  since  it 
is  not  presently  in  the  source  data,  but  it  may  be  the  only  real  way  to  improve  performance  in  this 
critical  subtask. 

Effectivity  codes  are  a  dynamic  characteristic  of  an  A/C,  yet  these  could  not  be  updated 
easily.  Since  much  of  this  information  is  available  in  CAMS,  it  would  be  worthwhile  to 
investigate  the  possibility  of  periodically  requesting  this  information  from  CAMS  and  updating 
the  state  table  automatically  to  reflect  these  changes. 

The  authors  of  the  electronic  technical  data  must  have  an  understanding  of  how  the 
presentation  system  operates  and  the  rules  it  uses  to  process  the  data.  Some  data  that  was  legal 
according  to  the  authoring  Style  Guide  caused  problems  or  did  not  work  as  expected.  Good 
communication  between  the  authors  and  the  software  developers  will  ensure  that  a  change  in  one 
area  has  minimum  impact  on  other  areas. 


Programmatics 

The  work  on  this  program  was  performed  under  two  separate  contracts  with  two  separate 
government  agencies.  As  a  result,  the  IMIS  program  did  not  have  direct  control  over  the 
authored  TO  data  or  the  TO  Present  software.  Differing  interpretations  of  requirements  for  data 
formats  and  software  between  the  two  separate  contractors  were  difficult  to  resolve.  User 
interface  differences  were  apparent,  transitions  between  the  two  systems  were  noticeable,  and  the 
conclusion  of  the  memorandum  of  agreement  resulted  in  the  IMIS  contractor  having  no  access  to 
TO  Present  source  code,  essentially  freezing  the  as-delivered  software  with  its  known  problems 
without  any  possibility  of  correction.  This  could  be  solved  by  ensuring  that  a  single  agency 
oversees  the  development  of  all  software  and  data  associated  with  the  program,  and  that  detailed 
specifications  documenting  the  interfaces  be  developed  and  followed. 

Establishment  of  the  Maintenance  Review  Panel,  with  its  composition  of  government  and 
contractor  personnel  with  active  duty  Air  Force  maintenance  experience,  was  extremely  valuable. 
This  group  focused  on  the  needs  of  the  ultimate  user  and  allowed  a  system  to  be  designed  and 
built  which  kept  these  needs  in  the  foreground. 

There  was  considerable  turnover  in  personnel  over  the  life  of  the  program,  for  both  the 
government  and  the  contractors.  It  is  important  to  maintain  continuity  of  key  resources  (or  at 
least  ensure  there  is  adequate  time  to  communicate  previous  decisions  and  agreements)  to 
minimize  changes  of  direction. 

Thorough  documentation  of  meetings,  conference  calls,  and  action  items  helps  to 
facilitate  effective  program  management.  As  documentation  improved  over  the  course  of  the 
program,  our  communication,  tracking  of  schedule  items,  resolution  of  action  items,  and  overall 
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program  management  also  improved.  Regularly  scheduled  conference  calls  were  also  beneficial, 
especially  in  a  dynamic  environment  where  face-to-face  meetings  were  impractical. 

Maintaining  schedule  milestones  with  the  necessary  level  of  detail  provides  visibility  into 
current  status,  for  both  the  contractor  and  the  government.  Early  schedules  lacked  the  detail  to 
allow  either  party  to  adequately  predict  schedule  impacts,  while  the  more  detailed  schedules 
developed  later  in  the  program  were  met  with  a  much  higher  success  rate.  It  is  also  important  to 
establish  the  appropriate  cost  reporting  documentation  corresponding  to  these  schedules. 

The  lack  of  a  contractual  requirement  for  Level  3  hardware  drawings  resulted  in 
differences  between  the  PMAs  built  by  GDE  Systems  and  the  PMAs  built  by  Computer  Sciences 
Corporation  (CSC).  The  hardware  documentation  provided,  which  met  all  contractual 
requirements,  was  inadequate  to  allow  a  second  contractor  to  build  identical  PMAs. 

Requirement  traceability  was  complicated  by  the  lack  of  a  System  Design  Document, 
which  would  have  captured  all  the  requirements  implemented  for  the  demonstration  system. 
Some  of  the  IMIS  documentation  captured  the  full-up  set  of  requirements,  while  other  documents 
captured  only  those  requirements  which  were  implemented.  If  a  System  Design  Document  had 
been  developed,  proper  and  meaningful  traceability  throughout  all  the  documentation  could  have 
been  established.  Any  changes  to  these  requirements  would  need  to  be  promptly  reflected  in  the 
contract. 

A  rigorous  and  well-defined  test  program  must  be  established  to  provide  the  government 
with  confidence  that  the  delivered  system  will  meet  the  requirements.  Acceptance  test 
procedures  which  are  used  to  verify  requirements  should  not  be  viewed  as  procedures  to 
demonstrate  all  functionality.  If  the  proper  test  environment  is  set  up  and  monitored,  the 
acceptance  test  process  can  be  very  effective. 


STRATEGIES  FOR  FURTHER  IMPLEMENTATION 

As  shown  by  the  results  described  in  the  second  section  of  this  document,  the  IMIS 
system  evaluated  at  Luke  AFB  was  successful  in  demonstrating  the  IMIS  concept  and  in  meeting 
the  objectives  identified  in  the  IMIS  Operational  Concept  Document.  The  next  logical  step  is  to 
apply  these  concepts  across  all  Air  Force  weapon  system  platforms  and  to  make  the 
implementation  generic  enough  so  that  it  can  be  applied  to  any  maintenance  environment 
(vehicles  and  communication  systems,  for  example). 

The  most  straightforward  example  of  extending  the  IMIS  concept  to  another  weapon 
system  platform  would  be  for  the  development  of  a  new  weapon  system.  The  requirements  for 
an  IMIS  system  would  need  to  be  considered  in  the  specification  for  the  weapon  system.  This 
would  ensure  that  the  appropriate  on-aircraft  capabilities  and  interfaces  are  developed,  and  that 
data  is  authored  to  support  the  weapon  system  maintenance. 
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In  the  case  of  an  existing  weapon  system  platform,  a  number  of  factors  must  be 
considered.  These  include  the  bases  at  which  the  weapon  system  is  deployed,  the  legacy 
databases  with  which  the  system  will  interact,  and  the  type  of  interface  with  the  weapon  system. 
It  was  found  that  Air  Force  policies  and  procedures  can  be  implemented  differently  from  one 
base  to  another;  consequently,  a  thorough  evaluation  of  current  implementations  at  these  bases  is 
needed  to  attempt  to  standardize  them  to  the  extent  that  a  single  system  can  be  developed  and 
used.  Different  bases  may  also  utilize  different  legacy  databases  (e.g.,  CAMS  vs.  Tactical 
Interim  CAMS  and  Reliability  and  Maintainability  Information  System  (REMIS)  Reporting 
System  (TICARRS)),  to  suit  their  unique  mission  needs. 

The  extensive  hardware  requirements  to  support  an  Air  Force-wide  implementation  of 
IMIS  will  result  in  additional  investigations  into  the  type  of  hardware  required.  Sun  workstations 
and  customized  PMAs  may  not  be  the  correct  answer  when  considering  the  acquisition  and 
development  costs  against  the  cost  of  utilizing  existing  resources  and  COTS  portable  devices;  for 
example,  a  486-based  system  would  allow  utilization  of  existing  hardware  at  most  bases. 

While  certain  aspects  of  the  system  are  dependent  on  the  weapon  system  or  base,  the  core 
of  the  Maintenance  Data  Collection  (MDC)  functionality  is  standard  and  independent  of  these 
variables.  By  migrating  this  MDC  functionality  to  allow  more  widespread  use  early  in  the 
acquisition  process,  additional  insights  can  be  gained  into  the  resulting  efficiency  improvements 
and  cost  reductions.  This  could  be  important,  since  the  activity  of  quantifying  the  cost  savings  of 
MDC  functions  as  implemented  using  IMIS  has  not  been  performed. 

In  summary,  the  Armstrong  Laboratory’s  IMIS  program  has  successfully  demonstrated 
the  IMIS  concept  and  has  produced  a  set  of  specifications  which  have  been  used  by  system 
implementers  for  the  F-16,  F-22,  C-17,  and  B-2.  The  information  gathered  from  this  effort  will 
be  a  factor  in  the  Integrated  Maintenance  Data  System  (IMDS)  efforts. 
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ACRONYMS 


A/C 

Aircraft 

AFB 

Air  Force  Base 

AFSC 

Air  Force  Specialty  Code 

AFTO 

Air  Force  Technical  Order 

AIP 

Aircraft  Interface  Panel 

APG 

Airplane  General 

BIT 

Built-In-Test 

CAMS 

Core  Automated  Maintenance  System 

COTS 

Commercial  Off-the-Shelf 

CSC 

Computer  Sciences  Corporation 

ETIC 

Estimated  Time  In  Commission 

FMC 

Fully  Mission  Capable 

FRM 

Fault  Reporting  Manual 

FS 

Fighter  Squadron 

GUI 

Graphical  User  Interface 

IMDS 

Integrated  Maintenance  Data  System 

IMIS 

Integrated  Maintenance  Information  System 

IPB 

Illustrated  Parts  Breakdown 

JCN 

Job  Control  Number 

LRU 

Line  Replaceable  Unit 

MDC 

Maintenance  Data  Collection 

MESL 

Minimum  Essential  Subsystem  List 

MIW 

Maintenance  Information  Workstation 

MML 

Memory  Module  Loader 

MOC 

Maintenance  Operations  Center 

NASA 

National  Aeronautics  and  Space  Administration 

NMC 

Not  Mission  Capable 

NMCS 

Not  Mission  Capable  -  Supply 

OFP 

Operational  Flight  Program 

OIC 

Officer  in  Charge 

PMA 

Portable  Maintenance  Aid 

PMC 

Partially  Mission  Capable 

PWC 

Performing  Work  Center 

QRL 

Quick  Reference  List 

REMIS 

Reliability  and  Maintainability  Information  System 

RF 

Radio  Frequency 

SBSS 

Standard  Base  Supply  System 

SCO 

Santa  Cruz  Operation 

SQL 

Structured  Query  Language 

TICARRS 

Tactical  Interim  CAMS  and  REMIS  Reporting  System 

TLX 

Task  Load  Index 

TO 

Technical  Order 

TO  Present 

TO  Presentation 
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WCE 

WDC 

WUC 


Work  Center  Event 
When  Discovered  Code 
Work  Unit  Code 
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